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Overview 


The New York City P-TECH Grades 9-14 schools represent an education model that ties together the 
secondary, higher education, and workforce systems as a way to improve outcomes in both domains. 
The distinguishing feature of the P-TECH 9-14 model, as it is referred to in this report, is a partnership 
between a high school, a local community college, and one or more employer partners that focuses on 
preparing students for both college and careers — not one or the other — within a six-year timeframe. 


Education and workforce development are traditionally seen as separate spheres of influence with 
multiple transition points that students have been left to navigate largely on their own (for example, 
high school to postsecondary, and postsecondary to the workforce). P-TECH 9-14 is designed to seam- 
lessly assist student navigation of those points — supporting student success and mitigating the po- 
tential for students to fall through the cracks. P-TECH 9-14 schools collaborate with local colleges to 
provide students with an opportunity to earn a high school diploma (within four years) followed by a 
cost-free, industry-recognized associate’s degree. During the six-year program, employer partners 
support P-TECH 9-14 schools by providing students with work-based learning experiences such as 
internships, mentoring, and job shadowing. By design, the P-TECH 9-14 model offers students the 
opportunity to participate in focused and accelerated high school pathways, early college, and career- 
focused activities. 


This study offers initial impact and implementation findings from the first rigorous evaluation of the 
model, evaluating the first seven P-TECH 9-14 schools that opened in New York City. The study 
leverages the random lottery process created by the New York City High School Admissions System 
to identify impacts. The majority of the students in the sample who participated in the admissions 
lotteries were academically below proficiency in both math and English language arts (ELA) prior to 
entering high school. 


Key Findings 


e Students’ high school coursework and New York State Regents exams are accelerated, and all 
schools focus on career and technical education (CTE) programs — classes that teach students 
specific workplace skills aligned with the labor market and “soft skills” such as good work habits 
and interpersonal skills. 


e College coursework begins largely in tenth grade and the pacing and progress of course taking 
varies by student. The degree pathways are designed to complement the high school CTE course- 
work and lead to credentials toward specific careers. 


e The specific work-based opportunities available, such as workplace visits, job shadowing, and 
internships, and levels of participation differed across schools. 


e P-TECH 9-14 students earned more total credits than students in other schools, with results driven 
by credit accumulation in CTE and other nonacademic subjects. These additional credits did not 
appear to come at the expense of earning academic credits. 


e At the end of two years of high school, 42 percent of P-TECH 9-14 students had passed the ELA 
Regents exam with a score qualifying them for enrollment in City University of New York 
(CUNY) courses, compared with 25 percent of comparison group students. By the end of three 
years, the gap was smaller but still favored P-TECH 9-14 students. 


e These pass rates indicate that more P-TECH 9-14 students were eligible to dual enroll in CUNY 
coursework in earlier years than their comparison group counterparts. 
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Executive Summary 


The New York City P-TECH Grades 9-14 schools represent an approach to career and technical 
education that aims to move well beyond the traditional vocational programs of the past. Initiated 
by IBM as a three-way partnership with the New York City Department of Education (NYC 
DOE) and the City University of New York (CUNY), the P-TECH 9-14 model is a multifaceted 
pathway program with a strong career focus that begins in high school and extends into college 
and the workforce.' These founding organizations opened the first P- TECH 9-14 school in Brook- 
lyn in 2011. Since then the model has attracted national and international attention. As of 2019 it 
was being used in 24 countries, with over 200 schools partnering with some 600 businesses. The 
State of New York has allocated approximately $40 million to fund P-TECH 9-14 expansion, 
making it a leader in P-TECH 9-14 student enrollment.* 


This report offers initial impact and implementation findings from the first rigorous eval- 
uation of the model and is funded by a grant from the federal Institute of Education Sciences. It 
presents the first look at the evaluation findings from the first seven P-TECH 9-14 schools that 
opened in New York City, provides background on the development of the model, and describes 
its high school, college, and work-based learning components. This report also discusses some of 
the ways in which the seven schools in the study vary in their implementation of the model. Fur- 
thermore, it provides information about the impact the P-TECH 9-14 model is having on students’ 
high school outcomes. The findings suggest that students are meeting the benchmarks that the 
P-TECH schools were designed to help them achieve. That may bode well for the future successes 
of these students, most of whom entered ninth grade with weak academic achievement in both 
English language arts (ELA) and math. 


The P-TECH 9-14 Model 


In the United States, education and workforce development are usually seen as separate spheres 
of influence, with multiple transition points that students have been left to navigate largely on 
their own (for example, from high school to postsecondary and from postsecondary to the work- 
force). The innovation of P-TECH 9-14 is that it is both an education model that ties together 
the secondary and higher education systems, and a workforce model aimed at bridging the 
school-to-work divide in order to improve outcomes in both domains. P-TECH 9-14 is designed 
to seamlessly support navigation of those systems, reducing the potential for students to fall 
through the cracks, particularly those who do not have additional sources of support. This is a 
distinguishing feature of the P-TECH 9-14 model: the partnership between a high school, a 
local community college, and one or more employer partners, all focused on preparing students 


'The New York City Department of Education refers to the model as the “Grades 9-14 Schools” and the 
City University of New York refers to the model as the “9-14 Early College and Career” or “NYC P-TECH” 
model. These terms distinguish this model from other early college model schools operating within New York 
City. For the purposes of this report, the authors refer to the model as “P-TECH 9-14.” 

Only two of the seven P-TECH 9-14 schools in this report operating in New York City receive state 
funding. 


for both college and careers — not one or the other — within a six-year timeframe. P-TECH 
9-14 schools collaborate with local colleges to give students an opportunity to earn a high 
school diploma (within four years), followed by a cost-free, industry-recognized associate’s 
degree. During the six-year program, employer partners support P-TECH 9-14 schools in var- 
ious ways, most commonly by providing students with work-based learning (WBL) experi- 
ences such as internships, mentoring, and job shadowing. Thus, the P-TECH 9-14 model offers 
students the opportunity to participate in focused and accelerated high school pathways, early 
college, and career-focused activities. 


What Does P-TECH 9-14 Look Like on the Ground? 


This report describes how the schools are developed, the P-TECH 9-14 high school experience, 
the six-year integrated education programming, college coursework, and work-based learning 
(WBL) opportunities for students, as illustrated in Figure ES.1. The findings are based on site 
visits, interviews, and a school leader survey. 


e The P-TECH 9-14 model relies on a partnership between a high school, a col- 
lege, and one or more employers. The NYC DOE and CUNY Early College 
Initiative offices support these partnerships and provide structured opportuni- 
ties for collaboration. 


e All of the components have been implemented at all seven study schools, but 
many of the specific elements vary. 


e Students’ high school coursework and the New York State Regents exams are 
accelerated, and all schools focus on career and technical education (CTE) 
programs (classes that teach students specific workplace skills such as archi- 
tecture and civil engineering or electrical engineering that are aligned with the 
labor market), as well as “soft” skills such as good work habits and interper- 
sonal skills. 


e College coursework begins largely in tenth grade; pacing and progress of 
course taking varies by student. The degree pathways are designed to com- 
plement the high school CTE coursework and lead to credentials for specific 
careers. 


e P-TECH 9-14 students participate in WBL activities such as workplace visits, 
job shadowing, and internships. The specific WBL opportunities available to 
students and the levels of student participation vary from school to school. 


Figure ES.1 


An Integrated Six-Year Progression 


The P-TECH 9-14 model starts in the ninth grade and 
integrates work-based learning opportunities and up to 
60 college credits at no cost to the student. 
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How Has P-TECH 9-14 Affected Students? 


The P-TECH 9-14 theory of action shows the benefit of students participating in a school model 
that creates a partnership between high school and college and employer partners from the fields 
of science, technology, engineering, and mathematics (STEM). This partnership provides oppor- 
tunities for students to participate in CTE coursework and offers early exposure to college readi- 
ness exams (in this case, the Regents exams), as well as work-based learning opportunities. These 
activities should increase attendance, Regents exam pass rates, and college credit-earning during 
high school (dual enrollment); fill skills gaps; decrease the need for postsecondary remedial clas- 
ses; and improve students’ preparedness to meet employer needs. In the long term, increases are 


expected in high school graduation rates, college enrollment and college credits earned, associate 
degree receipt, and rates of employment, as well as a reduction in the cost of higher education 
attainment. 


This report examines the early impacts of the P-TECH 9-14 model on student outcomes 
in their first three years of high school, including course credit accumulation in academic and 
CTE-related courses, Regents exam attempts and pass levels, and attendance. Given the elements 
of the model, one would expect P-TECH 9-14 students to take more CTE courses than other 
students, as well as to attempt to pass Regents exams with CUNY-qualifying scores earlier than 
their peers, in order to be eligible to take college level or dual-enrollment courses that are part of 
the model. 


This study used admissions lotteries created by the New York City High School Appli- 
cation Processing System (HSAPS) to form two groups of comparable students, akin to a random 
assignment study; the group that won admission to a P-TECH 9-14 school is referred to as the 
program group, while those who did not win a seat in one are referred to as the comparison group. 
This means that participation in the P-TECH 9-14 program caused the impact results discussed 
in this report. Overall, students in the study sample were mostly Black and Hispanic, from lower 
income neighborhoods, and underprepared academically for high school. Approximately 70 per- 
cent of the students were below proficient in eighth grade ELA and more than 70 percent were 
below proficient in eighth-grade math. 


The report provides evidence that the P-TECH 9-14 students in the study sample are 
achieving positive outcomes compared with a group of similar students in other schools and that 
the P-TECH 9-14 schools are, on average, setting students up to accomplish the milestones that 
the model is designed to help them achieve. 


e By the end of both the second and third years of high school, P-TECH 9-14 
students earned more total credits than students in other schools, with 
results driven by credit accumulation in CTE and other nonacademic 
subjects. By the end of three years in high school, P-TECH 9-14 students had 
earned an average of two additional credits more than students in the compar- 
ison group. These additional credits did not appear to come at the expense of 
earning academic credits, the accumulation of which remained statistically 
equivalent between groups for all three years of high school. 


e Students in P-TECH 9-14 schools were much more likely to earn nonac- 
ademic credits in work-based learning, technology, engineering, and hu- 
man service subjects. They earned fewer credits in arts, physical education, 
and health. The subjects where students accumulated more credits are aligned 
with the career and industry themes associated with the seven P-TECH 9-14 
schools. 


e P-TECH 9-14 students attempted more Regents exams than the compar- 
ison students in other schools, and a higher percentage of them passed the 


ELA Regents exam with a score qualifying them for enrollment in CUNY 
coursework in each of the first three years of high school. At the end of two 
years of high school, 42 percent of P-TECH 9-14 students had passed the ELA 
Regents with a CUNY-qualifying score, compared with 25 percent of compar- 
ison group students. By the end of three years, the gap was smaller but still 
favored P-TECH 9-14 students. 


e The CUNY-qualifying-score pass rates in ELA indicate that more P-TECH 
9-14 students were eligible to dual enroll in CUNY coursework in earlier 
years than their comparison group counterparts, which is in line with the early 
college aspect of this school model. 


These early findings are encouraging and indicate that P-TECH 9-14 schools are having 
a positive effect on students’ potential for transitioning to college and career settings. In particular, 
the increased accumulation of CTE and other nonacademic credits suggests that students are get- 
ting greater levels of career-related exposure than students in other schools. Furthermore, the var- 
ious types of CTE courses being offered indicate that these schools are providing different and 
potentially more modern career experiences. They are helping students succeed in ways that do 
not appear to come at the expense of earning academic credits. Students are doing better at accu- 
mulating credits in fields related to the careers the P-TECH schools aim to prepare them for. 


The positive findings on ELA Regents exam pass rates both at CUNY-qualifying levels 
and in earlier years than for comparison group students indicate that the P-TECH 9-14 model is 
successfully preparing students to take advantage of dual-enrollment opportunities at earlier 
points in high school. In addition, P-TECH 9-14 students ended their third year of high school 
with more Regents exams passed at the high school graduation level of 65 (a lower standard than 
a CUNY-qualifying score), and with better overall credit accumulation, leaving fewer credits to 
be earned in the fourth year of high school. This indicates that more P-TECH 9-14 students may 
be prepared to reach high school graduation than students in the comparison group. 


The P-TECH 9-14 model appears to be meeting the needs of academically low-perform- 
ing students, including those that entered the schools in eighth grade with the lowest levels of 
academic success. Overall, these interim findings are encouraging and provide evidence about 
how P-TECH 9-14 is supporting student success. Later reports from this study will focus on 
whether these findings foreshadow additional improvements in student outcomes related to high 
school graduation and college success. 


Chapter 1 


Introduction 


This is the first of several reports to be published by MDRC based on its evaluation of the New 
York City P-TECH Grades 9-14 model (hereafter referred to as P-TECH 9-14), an approach to 
career and technical education that aims to move well beyond traditional vocational programs of 
the past.' Initiated by IBM as a three-way partnership with the New York City Department of 
Education (NYC DOE) and the City University of New York (CUNY), the P-TECH 9-14 model 
is designed to be a multifaceted pathway program with a strong career focus that begins in high 
school and extends into college and the workforce. As of this publication, New York City operates 
nine P-TECH 9-14 schools. A distinguishing feature of the model is the partnership between a 
high school, a local community college, and one or more employer partners that focuses on pre- 
paring students for both college and careers — not one or the other — within a six-year timeframe. 
P-TECH 9-14 schools collaborate with local colleges to provide students with an opportunity to 
earn a high school diploma (within four years) followed by a cost-free, industry-recognized asso- 
ciate’s degree. During the six-year program, employer partners support P-TECH 9-14 schools in 
various ways, most commonly by providing students with work-based learning experiences such 
as internships, mentoring, and job shadowing. 


P-TECH 9-14 was conceived of as an antidote to a changing economy — one in which 
technology is placing new demands on workers, companies, and educational systems, and where 
companies can struggle to fill positions requiring technical knowledge, skills, and work experi- 
ence. The model also reflects a commitment by participating employers to provide skill-building 
opportunities that address equity in education and the workforce.’ In an increasingly bifurcated 
USS. labor market, well-paying jobs often require a college education, and workers with less for- 
mal education are often relegated to service-sector jobs.* This public-private partnership model 
aims to develop and provide opportunities for a middle-skills workforce that has more than a high 
school diploma but less than a four-year college degree. It seeks to close the global gap in science, 
technology, engineering, and math (STEM) skills, focused on but not limited to technology, en- 
gineering, health care, and advanced manufacturing.* 


Since the founding partners opened the first P-TECH 9-14 school in New York City in 
2011, the model has attracted national and international attention, including being mentioned by 


'The New York City Department of Education refers to the model as the “Grades 9-14 Schools” and the 
City University of New York refers to the model as the “9-14 Early College and Career” or “NYC P-TECH” 
model. These terms distinguish this model from other early college model schools operating within New York 
City. For the purposes of this report, the authors refer to the model as “P-TECH 9-14.” 

Business Roundtable (2019). 

Autor, Katz, and Kearney (2008). 

4Unlike in other P-TECH partnerships across the United States, advanced manufacturing is not a focal area 
in the New York City schools. 


President Barack Obama in his 2013 State of the Union address.* As of 2019, the model was 
operating in 24 countries and more than 200 schools, in partnership with 600 businesses. The 
State of New York has allocated approximately $40 million to expand the program, making it a 
leader in P-TECH 9-14 student enrollment.° This report examines the first seven P-TECH 9-14 
schools that opened in New York City and shows positive early findings on high school credit 
accumulation, most of which is driven by an increase in career and technical education (CTE) 
course taking — classes that teach specific workplace skills aligned with the labor market. The 
findings also show positive outcomes for students taking New York State Regents exams — 
standardized tests that New York City public school students must pass to earn a high school 
diploma. The report also provides insight into how the model operates across the various schools. 


Launching P-TECH 9-14 in New York City 


The P-TECH 9-14 model originated in 2010 when Stanley Litow — at the time the president of 
the IBM International Foundation and a former NYC deputy schools chancellor — approached 
then-NYC Schools Chancellor Joel Klein about creating a new educational model linking high 
school, college, and industry.’ That first school, called Pathways in Technology Early College 
High School (P-TECH), opened in Brooklyn the following year. The school was defined by six 
key design principles* that featured elements of quality CTE programs as well as “early college” 
programs that allowed high school students to graduate from high school with up to two years of 
college credit at no cost to students and their families: 


e A private-public partnership involving secondary education, a community col- 
lege, and one or more lead employer or industry collaborators 


e A six-year integrated education program in which high school and college 
coursework are seamlessly integrated, allowing students to begin taking col- 
lege courses during their high school years 


e Student exposure to work-based learning (WBL), including hands-on experi- 
ence in the form of internships and other opportunities 


e Open enrollment to promote access for all regardless of grade or testing re- 
quirements, with a focus on students from historically underrepresented 


groups 


e Student access to a cost-free associate’s degree recognized by industry, to help 
remove financial barriers associated with financing a college education 


5Kanter and Malone (2013). 

Only two of the P-TECH 9-14 schools operating in New York City receive state funding. 
7Kanter and Malone (2013). 

8See ptech.org. These are the design principals that IBM advocates for schools. 


e Employer partner commitment that P-TECH 9-14 graduates will be first in line 
for job interviews with that employer 


The P-TECH 9-14 model does not necessarily shorten the length of time it takes to obtain 
an associate’s degree; it is specifically designed to place students on a path to a college credential 
in a high-demand field within a six-year timeframe. The model also aims to prepare students for 
employment in “middle-skills” jobs — those that require more than a high school diploma but 
less than a bachelor’s degree. Initially conceived of as a way for IBM to address its future hiring 
needs, the model was also envisioned from the start as one that could expand on a large scale to 
include other employer partners.° 


Several factors made the original P-TECH school in Brooklyn a compelling model for its 
founders.'® On the industry side, IBM was finding it hard to fill positions that required STEM 
skills and was rethinking whether jobs traditionally requiring a bachelor’s degree could be filled 
by associate’s degree holders with appropriate training.''! At the same time, the wider U.S. labor 
market needed workers with strong content knowledge and technical skills in STEM fields that 
required some form of postsecondary education or training. CUNY, which has offered pathways 
to the middle class for generations of New York City’s low-income students, was poised to as- 
sume that postsecondary partner role; the university viewed the P-TECH 9-14 model as an itera- 
tion of its own successful early college initiatives. Finally, both CUNY and the NYC DOE had 
experience with CTE programs and small schools of choice,’ in addition to early college models, 
and were open to innovation. 


The timing couldn’t have been better. New models of CTE were gaining traction nation- 
wide as viable alternatives to the “four-year college for all” approach to preparing students for 
jobs with middle-class wages.!? Whereas traditional vocational education had been criticized for 
pushing low-income and minority students into remedial tracks that led to dead-end jobs, these 
new CTE models focused on preparing students for high-skill, high-wage work. The conditions 
were ripe for P-TECH 9-14’s expansion in New York City. 


Two years after the first school launched, NYC DOE and CUNY opened two new 
P-TECH 9-14 schools in 2013: Health, Education and Research Occupations Early College High 
School (HERO), and Energy Tech High School. A year later, Business Technology Early College 
High School (B-TECH), Manhattan Early College High School for Advertising (MECA), and 


°While the original program was aligned with IBM’s historical commitment to corporate social responsibil- 
ity and education improvement, the company did not intend to control its future expansion. Rather, IBM aimed 
to serve as an industry partner for one school in New York City, the first P-TECH school to open, which is one 
of the seven schools included in this evaluation. As P-TECH 9-14 has scaled nationally and internationally, 
founding partner IBM remains a strong champion and ambassador for the model. The company is still paired 
with the first P-TECH 9-14 school, in addition to several other schools outside of New York City that are not 
included in this evaluation. 


‘0 Kanter and Malone (2013). 

'ITBM eventually restructured some of its positions to accommodate those at the associate degree level. 

!2Small schools of choice are small, nonselective high schools in New York City. Bloom and Unterman 
(2013). 

Rosen, Visher, and Beal (2018). 


Inwood Early College for Health and Information Technologies opened their doors to their first 
class of ninth-grade students. In 2015, City Polytechnic High School of Engineering, Architecture 
and Technology (City Poly) transitioned from an already existing five-year, early-college high 
school to a P-TECH 9-14 school.'* These seven schools, the focus of this evaluation, are each 
partnered with an individual CUNY college and one or more employer collaborators representing 
technology, engineering, and health care.'> See Figure 1.1 for more details about the P-TECH 
9-14 schools included in this study and their partners. 


The Evaluation and this Report 


As noted above, the P-TECH 9-14 model includes elements of other large-scale high school re- 
form efforts, including small schools of choice and early college high schools. In addition, it 
builds on the career academies model, which assigns students to small learning communities, 
combines academic and technical curricula around a career theme, and establishes partnerships 
with local employers to provide WBL opportunities. All of these models have been rigorously 
evaluated and found to produce large, positive impacts on student outcomes, including high 
school graduation, college enrollment, and earnings.'° While some individual P-TECH 9-14 
schools have reported positive student outcomes, this study is the first to rigorously assess the 
impact of the model on student outcomes by comparing those of students who randomly won the 
opportunity to attend P-TECH 9-14 schools through the New York City high school admissions 
lottery with students who randomly lost the same opportunity. This is also the first study to con- 
duct systematic research focused on the model’s implementation and to estimate the cost of op- 
erating these schools. As the first study of its type, it will contribute to building reliable and ac- 
tionable research evidence about the efficacy of the P-TECH 9-14 model. 


This interim report focuses exclusively on early implementation and impact findings dur- 
ing students’ first three years of high school. Chapter 2 provides early insights into how the model 
operates and how it was implemented across the seven study schools during the 2017-18 and 
2018-19 school years. The implementation findings are based on a school leader survey and in- 
terviews the study team conducted with school administrators, staff, students, and partners. The 
implementation research was designed to answer questions about the context in which the schools 
operate, dosage (or exposure to the P- TECH 9-14 model), and fidelity (or alignment to the model). 


'4The CUNY Early College Initiative (ECI) oversees 10 early college high schools throughout the city be- 
yond the P-TECH 9-14 schools. These schools differ from the P-TECH 9-14 model in that they educate students 
in grades 6-12, 9-12, and 9-13; offer a general liberal arts curriculum rather than a STEM-focused career path; 
and do not necessarily have an employer partner. 

'S P_TECH 9-14 has continued to expand. As of this writing, NYC DOE has opened two additional schools 
and a third is in the planning phase. While the first seven schools are included in this study, these newer schools 
are not. In New York State, where millions of dollars have been invested in the scale-up effort, P-TECH 9-14 
represents an important policy priority. Forty-two P-TECH 9-14 schools are in operation at the state level. 

‘6Bloom and Unterman (2014); Kemple (2008); Berger et al. (2013); Edmunds et al. (2012). 
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New York City P-TECH 9-14 Schools and Partners 
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Chapter 3 presents findings from the impact study, which takes advantage of lotteries in NYC 
DOE’s high school choice admissions process to conduct an experimental study using retrospec- 
tive data for several cohorts of students enrolled in the study schools. The analyses look at the 
P-TECH 9-14 model’s impact on student outcomes during the first three years of high school, 
including New York State Regents exam attempts and passage, engagement in high school CTE 
and academic course work, and high school attendance. The report concludes with a discussion 
of policy implications and next steps in the evaluation. 


Chapter 2 


The New York City P-TECH 9-14 Model and 
Early Implementation Findings 


New York City’s P-TECH 9-14 model, which creates a partnership between a high school, col- 
lege, and employer partner, is aligned with the design principles described above.! Of particular 
note, all of the city’s P-TECH 9-14 schools are open enrollment, and students taking college 
coursework do not pay for classes or textbooks. This section describes the P-TECH 9-14 theory 
of action and how the schools are developed, as well as the high school experience, college 
coursework, and work-based learning (WBL) opportunities. It is based on findings from the study 
team’s site visits, interviews, and school leader survey. 


The P-TECH 9-14 schools’ theory of action shows the benefit of students participating 
in a school model based on a partnership between a high school, a college, and one or more em- 
ployer partners in the science, technology, engineering and math (STEM) fields. This partnership 
provides opportunities for students to participate in career and technical education (CTE) classes 
as a part of integrated high school and college coursework, early exposure to college readiness 
exams (in this case, the standardized New York State Regents exams, described in more detail 
below), and exposure to WBL opportunities. These activities should increase attendance, Regents 
exam pass rates, dual-enrollment credits, and students’ preparedness to meet employer needs and 
fill skills gaps, and decrease the need for postsecondary remedial classes. In the long term, 
increases in high school graduation rates, postsecondary enrollment and college credits earned, 
associate’s degree receipt, and rates of employment are also expected, as well as a reduction in 
the cost of higher education attainment. This report will explore early impacts on three indicators 
in the theory of action: high school credit accumulation toward graduation, Regents exam perfor- 
mance, and attendance. 


Figure 2.1 provides an overview of the P-TECH 9-14 model. As shown in the “High 
School” row of this figure, P-TECH 9-14 students’ experience includes taking state Regents ex- 
ams and CTE coursework beginning early in their high school career. As shown in the “College” 
row, students begin connecting with their school’s partner college very early in their high school 
career, and they can begin taking college classes as early as tenth grade, with their coursework 
becoming increasingly specialized over time. Finally, as shown in the “Career” row, students’ 
WBL activities shift from a focus on career exposure and awareness in the early grades to more 
intensive career exploration and training opportunities in the later grades. 


'P-TECH 9-14 schools in New York City do not require industry partners to commit to putting graduates 
first in line for jobs. 


Figure 2.1 
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Highlights from this chapter include: 


e Each P-TECH 9-14 school relies on a partnership between the high school, 
college partner, and employer partner(s). The New York City Department of 
Education (NYC DOE) and the Early College Initiative of the City University 
of New York (CUNY) support these partnerships and provide structured op- 
portunities for collaboration. 


Overall, while these components are being implemented in all of the study 
schools, there is also variation in many of the specific elements. 


e Students’ high school coursework and Regents exam taking are accelerated 
and all schools focus on CTE and “soft” skills. 


e College coursework begins largely in tenth grade, and the pacing and progress 
varies by student. The degree pathways are designed to complement the high 
school CTE coursework and lead to credentials tied to specific careers. 


e P-TECH 9-14 students participate in WBL activities such as site visits, job 
shadowing, and internships. The specific WBL opportunities available to stu- 
dents and the levels of student participation can differ from school to school. 


The high school, college, and WBL sections below focus primarily on common aspects 
of the model across the P-TECH 9-14 schools. Subsequent reports will examine dual enrollment, 
provide additional information on the schools’ high school, college, and career elements, and 
delve more deeply into variation both across and within the P-TECH schools. 


P-TECH 9-14 Partnerships 


P-TECH 9-14 schools require coordination and cooperation among three key partners: high 
schools and their school districts, college institutions and systems, and partner employers. Suc- 
cessful implementation hinges on a commitment to change, and each group brings specific histo- 
ries, cultures, standards, approaches, and goals to bear on the partnership. In New York City, the 
P-TECH 9-14 school development process is co-led by CUNY’s Early College Initiative (ECT) 
and NYC DOE’s Office of Postsecondary Readiness (OPSR). Together they identify employer 
partners and provide support and oversight of model implementation. 


During the startup of a school, ECI and OPSR help analyze labor market trends to identify 
potentially sustainable career pathways, map out necessary skills, convene partners, and assist in 
hiring the school principal. ECI and OPSR also work with individual P-TECH 9-14 partner teams 
to create the scope and sequence of high school and college course offerings. 


After startup, P-TECH 9-14 schools meet regularly with ECI and OPSR staff represent- 
atives, who conduct professional development gatherings for school staff members, participate in 
steering committees made up of representatives from each partner organization, and convene 
principals from across the schools at least twice a year. Each college and employer partner work 
with the high school to monitor implementation and make adjustments as necessary, so students 
have as cohesive an experience as possible. In addition to contributing to the vision of the school, 
the employer partners also provide work-based learning supports to the students, which may in- 
clude providing professional mentors, giving workplace tours, advising on class projects, and 
offering paid internships. Though IBM is an advocate for the model and is partnered with the first 
New York City P-TECH 9-14 school, its affiliation does not involve providing monetary support, 
oversight or quality control for P-TECH 9-14 schools more broadly. 


P-TECH 9-14 High School Experience 


P-TECH 9-14 students, like other public high school students in New York City, are expected to 
meet NYC DOBP’s high school requirements within four years for on-time graduation, in order to 
receive a high school diploma.* However, there are a few distinguishing features specific to the 
P-TECH 9-14 experience, including early Regents exam taking, CTE classes, and soft skills in- 
struction, as described below. 


Accelerated High School Schedule 


P-TECH 9-14 students are on an accelerated timeline with their coursework, and their 
high school classes are frontloaded. Principals reported that they wanted to make sure that stu- 
dents would not become overwhelmed with college classes in their later high school years and be 
unable to complete their standard high school requirements on time. Although some students may 
finish their required high school coursework early, P-TECH 9-14 students who have met New 
York City’s requirements still receive their diplomas at the end of twelfth grade. 


Given the many layers of the high school experience, students need advising and guid- 
ance support. Many of the P-TECH 9-14 schools in this study also provide a summer bridge 
experience, to get their students ready for high school in the summer before ninth grade. Principals 
reported needing extra guidance and advising staff in order to support students through the mul- 
tifaceted P-TECH 9-14 program. Four of the schools have students meet with counselors to start 
discussing college and career in ninth grade, while two start in tenth grade, and one school varies 
its starts by student. The schools are still exploring the best way to create cohesive, individualized 
college and career plans for each student. 


Regents Exams 


As a core element of the P-TECH 9-14 model, New York State Regents exams — a 
suite of standardized tests covering a range of subjects — are taken early. Subjects include 
English Language Arts (ELA) and various math, science, and social studies tests, with core 
subject exams required for high school graduation. Statewide, the exams are offered three times 
a year, and students may take them multiple times until they pass. However, there is no state 
policy specifying when students must take particular exams, which gives schools flexibility 
over when they encourage their students to take them. In general, CUNY requires P-TECH 9- 
14 school students to achieve a college-ready score (which is higher than the score required for 
high school graduation) in ELA and one math subject before they can begin taking college 
classes toward a degree, although some CUNY campuses allow students to take one entry-level 


?Students in the New York City school system can graduate with either a Regents Diploma or an Advanced 
Regents Diploma, both of which require 44 credits to graduate. Students earning a Regents Diploma need to pass 
five Regents exams, while students aiming for an Advanced Regents Diploma need to pass nine Regents exams 
(New York City Department of Education, 2020). 
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course before meeting the college-ready benchmark.’ In order to prepare students to begin dual- 
enrollment coursework at CUNY, P-TECH 9-14 students can begin taking Regents exams as 
early as the summer before ninth grade and are encouraged to have attempted them by the end 
of their tenth-grade year; the timing of students’ first Regents exam varies across these schools. 


Career and Technical Education 


The P-TECH 9-14 high schools are all considered CTE-designated high schools by the 
NYC DOE, as all students are enrolled in a CTE program of study.’ Students must participate in 
one of their school’s specified CTE pathways. A CTE pathway is a scope and sequence of classes 
that build technical skills to prepare for a specific postsecondary focus or career. Almost all 
P-TECH 9-14 schools in the city require students to take more than four sequential CTE courses, 
which build on each other, in order to complete the average CTE pathway. These focused path- 
ways and high school course completion requirements indicate the depth of the CTE experience 
offered at P-TECH 9-14 schools.° 


Soft and Professional Skills 


In addition to CTE coursework, there is also an explicit focus on “soft skills” such as 
social-emotional skills, staying organized, and working well with a team.° Related professional 
skills such as writing a resume and interviewing for a job are also taught in the P-TECH 9-14 
school model and are meant to complement the WBL experiences, preparing students to obtain 
work and perform satisfactorily in the workplace. Almost all of the schools teach these skills in 
advisory periods for which students may or may not earn credits; the offerings vary across schools 


3In New York State, a passing score on these exams for a Regents high school diploma is 65 or higher 
(Higher Education Services Corporation, 2019). However, CUNY has higher score requirements for ELA and 
math Regents exams in order to demonstrate college readiness and qualify to skip remedial classes before starting 
an associate’s degree program; the score requirements vary by exam and by the year of the study. The content of 
the Regents exams also changed during the course of this study. In June 2014, the ELA and algebra exams were 
changed to incorporate the Common Core Standards. Those standards were also added to geometry (June 2015) 
and algebra 2 (June 2016). The City University of New York (2019). 

4A CTE program of study is a sequence of classes based on industry and New York State Education De- 
partment (NYSED) learning standards to develop technical skills for a specific career pathway. A CTE program 
of study includes WBL experiences, a three-part technical assessment, and an articulated agreement with a post- 
secondary institution for advanced standing or college credit for students who successfully complete the technical 
assessment. NYSED has an endorsement process in place to evaluate and approve formal CTE programs of 
study offered in New York state schools. Students who complete all aspects of a NYSED-approved CTE pro- 
gram of study are eligible to graduate with a CTE-endorsed diploma and use the program’s technical assessment 
as one of the alternate assessment options for graduation. Two of the seven P-TECH 9-14 schools currently have 
NYSED-approved CTE programs of study and the others are in the process of applying for program approval. 

*By comparison, the federal Perkins V act, which governs CTE, only requires two sequential courses to be 
considered a “CTE concentrator.” 

®Soft skills may also be referred to as twenty-first-century skills, professional skills, or career skills. Soft 
skills may encompass social-emotional skills related to communication and social interactions, as well as skills 
such as interviewing, resume writing, and project management that are meant to prepare students for college and 
career. 
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and school years.’ Five schools also have classes explicitly designed to build college and career 
skills and three have integrated this skills training into academic and CTE classes. Two schools 
also have WBL coordinators who have created soft and professional skills curricula, and most 
have received input from their employer partner on which skills students should be taught before 
they enter the workplace. 


P-TECH 9-14 College Coursework 


P-TECH 9-14 students can begin taking college coursework free of charge before earning their 
high school diploma at the end of twelfth grade. In most cases, P-TECH 9-14 students will earn 
degrees that already exist at their partner CUNY school; in at least one case, however, the partner 
college created a new degree pathway after discussions with the P-TECH school and the employer 
partner about the growth of a specific labor demand within the industry. Each high school has a 
college liaison who splits time between the high school and college campuses and provides a 
variety of supports including course scheduling and advising, connecting students to tutoring op- 
portunities, and advocating on students’ behalf with professors. 


College Opportunities 


All seven schools reported that P-TECH 9-14 students can begin taking college classes 
in tenth grade. As described above, the first class often does not require the CUNY Regents exam 
benchmarks and acts as an introduction to college coursework. For later classes taken toward a 
student’s major, in addition to meeting the CUNY Regents exam benchmarks, some schools also 
require students to have a good attendance record, a counselor recommendation, or a minimum 
GPA before they can take college classes. These additional requirements are often developed in 
collaboration with the college partner.® If students do not meet the required benchmarks, they 
continue with their regular high school coursework. Although there are multiple opportunities to 
meet benchmarks and begin taking college courses, schools did report that there are students who 
never become eligible for early college coursework; this may highlight a tension between the 
open-enrollment and early-college-ready elements of the model. 


Once college courses have begun, coursework and sequences are pre-set within the de- 
gree pathways, of which all of the schools offer at least two. Students are working mostly toward 
Associate of Applied Science degrees (AAS), although three schools also offer Associate of Sci- 
ence degrees (AS), and one offers only AS degrees. These degrees complement the high school 
CTE coursework, and the combined scope and sequence builds students’ experience and creden- 
tials toward specific careers. Students begin taking college courses exclusively with other 
P-TECH 9-14 students. At almost all of the P-TECH 9-14 schools, students begin with college 
instructors who come to their school to teach the classes. As students advance, they take an 


TAdvisory periods are dedicated times for teachers to meet with small groups of students and advise on a 
range of aspects, including social-emotional needs, academic best practices, and college and career planning. 

8Interviews with school staff suggest that the CUNY Regents benchmarks—not the additional requirements 
schools put in place such as attendance and recommendation letters—are the primary impediment to students 
taking college pathway coursework. 
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increasing number of classes on the college campus that may include a mix of P-TECH 9-14 and 
other students. Regardless of the location and mix of students, study schools reported that the 
content and expectations are the same as in a standard college class. 


Students who are dual-enrolled in this way begin taking their early college classes sim- 
ultaneously with their high school coursework, rather than completing all of their high school 
credits first.? Students’ progress and the pacing of their college course-taking throughout their 
time as a P-TECH 9-14 student varies. For example, one student might take an introductory col- 
lege course before passing the Regents exam but may need to refrain from taking major-specific 
courses before passing the requisite exams. Another student might begin college courses but start 
to fall behind in their high school classes, requiring them to stop taking the college courses in 
order to focus more fully on high school graduation requirements. A small number of students 
have been able to complete the high school and college coursework in less than the six years of 
the model and receive both their high school and college diplomas in that time. The fluidity of 
the experiences underscores the importance of P-TECH 9-14 students having advisors: Students 
frequently need individualized advising to determine the combination of high school and college 
coursework that will best meet their needs. 


P-TECH 9-14 Work-Based Learning Activities 


One of the distinguishing characteristics of P-TECH 9-14 schools is the focus on providing 
students with opportunities to engage in WBL activities that are aligned with their high school 
and college coursework. In this report, WBL can refer to career-related activities both in the 
school building and at the workplace, though all involve direct involvement with industry em- 
ployers. Each school’s employer partner plays an active role in helping to culttvate WBL op- 
portunities. An industry liaison, employed by the employer partner, spends at least some of 
their time working in this capacity, sometimes visiting the school to collaborate and plan with 
school staff. While all schools reported that their founding employer partner provided at least 
some WBL opportunities, the schools also had other sources of support for these activities. 
Specifically, staff members from nearly all the schools actively sought connections with em- 
ployers outside the program to create a more robust set of WBL opportunities for their students 
and reported that staff from the NYC DOE and/or CUNY helped identify or facilitate internship 
opportunities. Additional employer partners can be necessary to host the full number of students 
participating in internships and provide different types of opportunities for students with 
broader sets of interests. See Box 2.1 for additional details on the role of employer partners. 


*For purposes of this report, “dual-enrolled” means that a student is taking college classes and earning 
college credit. It does not mean that they are earning high school credit for the college credits that they 
earn. 
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Box 2.1 


Role of Employer Partners 


Employer partners are an essential part of the P-TECH 9-14 model. Founding employer partners 
work with the New York City Department of Education (NYC DOE) and the City University of 
New York (CUNY) from the beginning ofa school’s start-up. Having been identified as a com- 
pany in an industry that can provide a middle-income job ladder, the employer partner collabo- 
rates with NYC DOE and CUNY to work backward: first, identifying the skills and credentials 
needed for their industry and then building a high school and college scope and sequence to 
teach students those skills and provide them with the necessary credentials. Companies may 
partner with a P-TECH 9-14 school for a variety of reasons, including wanting to create a pipe- 
line of talent for their company or industry or to give back to the community. It is important that 
the right representative from the company takes part in these early conversations. For example, 
corporate responsibility representatives may be able to initiate the relationship, but the human 
resources department must be included to ensure that there is a realistic path to a career awaiting 
students at the end of the model. 


Once the school is open, employer partners provide work-based learning opportunities to stu- 
dents, first exposing them to the workplace and then providing opportunities for participation. 
Companies commit at varying levels depending on their own capacity; activities can range from 
weekly workplace visits to hosting students for summer internships. Though it is one of the 
original P-TECH 9-14 design principles, not all employer partners can commit to graduates be- 
ing first in line for job interviews. Employer partnerships are most successful when the school 
and employer have shared expectations for the relationship. Regular steering committee meet- 
ings with the high school, college, and employer partners are designed to keep these expectations 
current and the communication channels clear. 


WBL Opportunities and Sequencing 


The WBL activities most commonly available at P-TECH 9-14 schools include work- 
place visits, job shadowing, internships, and guest speaking and mentoring by an industry pro- 
fessional. Early in students’ high school careers, WBL focuses mostly on providing exposure 
to work in a particular industry and activities that build awareness of industry roles. As dis- 
cussed below, this is followed in the later grades by more intensive and hands-on career explo- 
ration and training opportunities. Most WBL is embedded within CTE classes, though there are 
also stand-alone WBL experiences for which students may earn credits, also discussed below. 
For example, a P-TECH 9-14 school may require an engineering class with a project-based 
learning component in which students must solve an engineering problem as a team and present 
their final product to industry professionals. Larger one-off events, such as a workplace visit or 
a professional speaker, also often happen during CTE class time, which teachers may supple- 
ment with activities such as reflection discussions. Throughout these activities, students get the 
opportunity to practice soft skills such as adaptability, critical thinking and problem solving, as 
well as the chance to use skills such as resume writing and interviewing in a real-world setting. 
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Career Exposure and Awareness 


P-TECH 9-14 students’ earliest WBL opportunities include activities such as career- 
focused speakers and workshops. In most of the schools, a significant number of ninth-graders 
have the opportunity to visit a job site to learn about the company’s work and the types of jobs 
available. Visits are primarily to the school’s designated employer partner, but some schools 
reported that students also visit other companies. In three schools, tenth-graders also participate 
in workplace visits as frequently as ninth-graders. 


All P-TECH 9-14 schools reported bringing in career-focused speakers as part of the 
early career exploration activities. In many of the schools, students also engage in projects and/or 
present their work to industry professionals. For example, at one school, students have the oppor- 
tunity to visit the employer partner as often as once a week for activities with employees on site. 
At another school, students develop business plans and present them to employees from the part- 
ner company. 


Industry mentoring is another component of students’ WBL activities at most P-TECH 
9-14 schools. In some, students participate in either one-on-one or small- group mentoring be- 
ginning in ninth grade, and participation in mentoring continues for multiple years (though stu- 
dents do not typically have the same mentor for more than one year). In three of the schools, 
opportunities for mentoring have been less consistent: One school offers mentoring to a subset 
of female students in engineering to provide them with female role models in the industry; 
another school is in the process of piloting a new mentoring program; and a third school has no 
mentoring program. 


Career Exploration and Training 


As students progress in their high school careers, they may also engage in more inten- 
sive WBL activities such as paid internships and job shadowing, in which they observe an in- 
dustry professional doing their job.'!° However, most schools are unable to provide these op- 
portunities for all of their students. Specifically, while one school reported that most of their 
ninth- to eleventh-graders engage in job shadowing, at other schools, job shadowing opportu- 
nities are more limited. 


Finally, paid internships, which are often considered the pinnacle of WBL activities, are 
available in some form at all P-TECH 9-14 schools. However, there is substantial variation in the 
levels of student participation across schools, both during the summer and the school year, as 
reported by principals. Of the seven P-TECH 9-14 schools, three reported that only a handful of 
students per grade participated in paid summer internships, though one of those schools said up 
to a third of their students participated in internships in the summer after twelfth grade in partic- 
ular. The remaining four schools reported greater levels of student participation, especially in the 
summers after eleventh and twelfth grades, with roughly one quarter to one half of eleventh- and 
twelfth -graders participating in paid summer internships. Internships during the school year were 


'0Schools also continue to provide students with the exposure-focused WBL opportunities described above. 
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also most frequently available for eleventh- and twelfth-graders. All but one of the schools re- 
ported that students participated in paid internships during the school year. Three schools reported 
higher participation, with approximately 20 to 40 percent of eleventh- and twelfth-graders partic- 
ipating. These three schools also had high numbers of summer internships. Despite this variation 
in later grades, one consistent finding was that students in all seven schools were less likely to 
participate in paid internships in grades nine and ten than in grades eleven and twelve. 


While these data suggest that paid internships are available to many P-TECH 9-14 stu- 
dents, it is also clear that ultimately, many students do not participate in them. All schools have 
requirements for who is able to take part. Some are set by the school and may include a minimum 
GPA (set either by the school or employer), attendance and behavioral requirements, or a recom- 
mendation from a school staff member. Some requirements are set by the employer partner. For 
example, schools with health care employer partners require students to complete health screen- 
ings in order to be cleared to work, and schools with engineering employer partners require interns 
to be a certain age. The final report will include additional details on which students participate 
in various WBL activities and why, as well as the content of students’ internships and other WBL 
opportunities. 
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Chapter 3 


The Impact Study 


This chapter examines the early impacts of P-TECH 9-14 on students’ outcomes in their first 
three years of high school. The chapter begins by providing an overview of the study design — 
including how the New York City high school admissions system works, and how it provides an 
opportunity for rigorous evaluation — as well as the characteristics of students in the study. Spe- 
cifically, this study uses admissions lotteries created by the New York City High School Appli- 
cation Processing System (HSAPS) to form two groups of comparable students, akin to a random 
assignment study. 


The chapter discusses the effects of the P-TECH 9-14 model on students’ interim high 
school outcomes, including their course credit accumulation in both academic and career and 
technical education (CTE) and other nonacademic courses; Regents exam attempts and pass lev- 
els; and attendance within the first three years in high school. Given the elements of the model, 
one would expect P-TECH 9-14 students to take more CTE courses than other students, as well 
as to attempt to pass Regents exams with CUNY-qualifying scores earlier than other students, in 
order to be eligible for the dual-enrollment courses that are part of the model.' Impacts on high 
school graduation or postsecondary outcomes (including those related to dual enrollment) will be 
examined in subsequent reports. 


Overall, the report provides evidence that students in the P-TECH 9-14 study sample, 
more than two-thirds of whom entered ninth grade below a proficient level in both English lan- 
guage arts (ELA) and math, are, on average, accomplishing many of the milestones that the model 
is designed to help them achieve. In particular: 


e By the end of both the second and third years of high school, P-TECH 9-14 
students earned more total credits than students in other schools, with results 
driven by credit accumulation in CTE and other nonacademic subjects. By the 
end of three years in high school, P-TECH 9-14 students had earned an aver- 
age of two credits more than students in the comparison group. These addi- 
tional credits did not appear to come at the expense of earning academic cred- 
its, the accumulation of which remained statistically equivalent between 
groups for all three years of high school. 


e Students in P-TECH 9-14 schools were much more likely to earn nonacademic 
credits in work-based learning, technology, engineering, and human service 
subjects. They earned fewer credits in arts, physical education, and health. 


‘All data for the impacts section of this report come from the longitudinal file of New York City Department 
of Education (NYC DOE) administrative data records, housed and maintained by the Research Alliance for New 
York City Schools. 
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These subjects are aligned with the career and industry themes associated with 
the seven P-TECH 9-14 schools. 


e Ineach of the first three years of high school, P-TECH 9-14 students attempted 
more Regents exams than students in comparison schools, and more P-TECH 
9-14 students passed the ELA Regents exam with a score qualifying them for 
enrollment in CUNY coursework than comparison students. At the end of two 
years of high school, 42 percent of P-TECH 9-14 students had passed the ELA 
Regents with a CUNY-qualifying score, compared with 25 percent of compar- 
ison group students. By the end of year three, the gap was smaller but still 
favored P-TECH 9-14 students. 


e The CUNY-qualifying-score pass rates indicate that more P-TECH 9-14 stu- 
dents were eligible to dual enroll in CUNY coursework in earlier years than 
their comparison group counterparts, which is in line with the early college 
aspect of this school model. 


How the New York City High School Lottery Works 


In New York City, eighth-grade students apply to a centralized system to apply to high school. 
Students are allowed to rank up to 12 schools they are interested in attending, and an algorithm 
matches students to schools based on whether they meet the admissions criteria for a given school 
and the order of preference listed by the student. In some cases, when more students meet a 
school’s admissions priority categories than there are available seats in a school, a process similar 
to a lottery occurs in which some students randomly win admission to a particular school, while 
other students randomly lose the opportunity to attend the school. (For a more detailed description 
of this admission process, please see Appendix A.) During the admissions years covered by this 
study, all seven P-TECH 9-14 schools were oversubscribed, meaning there were random lotteries 
for at least a subset of seats in each of the schools. This random process for allocating seats pro- 
vides an opportunity for a natural experiment, akin to a random assignment study, in which two 
comparable groups are created. In this report, the group that won admission to a P-TECH 9-14 
school is referred to as the program group, while those who did not win a seat in one of the schools 
are referred to as the comparison group. 


Importantly, not all students who were assigned a seat in the P-TECH 9-14 schools were 
admitted by lottery. Each school establishes admissions priority categories: The first students ad- 
mitted are those who meet priority category one, followed by those who meet category two, and 
so on. Since P-TECH 9-14 schools do not have academic or attendance criteria for admissions, 
these categories include such things as whether students live in the same New York City borough 
as where the school is located, and whether the student made contact with the school through a 
high school fair or other means prior to applying.” In some cases, there were fewer students who 


These admission categories were part of a NYC DOE school designation known as “Limited Unscreened.” 
During the cohort years included in this study, all P-TECH 9-14 schools were Limited Unscreened. However, 
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met a school’s top one or two priority categories than there were seats for the entering ninth-grade 
cohort, so students who met those top criteria were admitted without a lottery. The remaining 
seats in lower-priority categories were then assigned by lottery. By the same token, if more stu- 
dents who met a school’s top priority categories applied, then all seats were allocated randomly. 
For purposes of this study, only the students who were placed in their first lottery process for 
school admission are included in the analytic sample. 


While this process provides a rigorous set of comparable groups of students, contextual- 
izing the findings presented here also requires understanding whether there are differences be- 
tween students admitted via lotteries and those who gained admission and enrolled in these 
schools via non-lottery placements. In this case, just over half (53 percent) of students who were 
offered admission to the P-TECH 9-14 schools during the study years were admitted via a lottery. 


In addition, while the students in the program group were concentrated in the seven 
P-TECH 9-14 schools, those students who did not win a seat in one of these schools ended up 
receiving admissions offers from one of a total of 399 other high schools across New York City, 
which is almost all other high schools in the city. One hundred of these schools were either dedi- 
cated CTE high schools or academic high schools offering some CTE programing. Thirty-eight 
percent of comparison group students were enrolled in the schools that offered some kind of CTE; 
the rest of the comparison group students attended academic high schools without significant 
CTE offerings. In addition, using data from the Office of Civil Rights, the report team found that 
in the 2015-2016 school year, approximately 26 percent of students in the comparison group at- 
tended schools that offered dual-enrollment or dual-credit experiences.* Overall, the total number 
of comparison students enrolled in any single non-P-TECH 9-14 school of any kind does not 
exceed 39 (or 2 percent of the comparison sample), and in many cases is as low as a single student. 
For these reasons, the comparison group can be thought of as “students who experienced any non- 
P-TECH 9-14 school in New York City.” This is a very broad comparison that represents a wide 
range of student high school experiences. 


Finally, because this study design is like a student level random assignment study, results 
should also be interpreted at the student level — that is, the estimates presented here are the av- 
erage impact for the average student who applied to any P-TECH 9-14 school and who was ad- 
mitted via first lottery between 2013-2017. 


NYC DOE eliminated the Limited Unscreened designation beginning in the 2019-20 school year, and all 
P-TECH 9-14 schools became what are known as Education Options schools, which have different admissions 
priorities. 

3For more information about how the students in the analytic sample compare to the overall population of 
P-TECH 9-14 admitted students, please see Appendix Table A.4. 

4Civil Rights Data Collection (CRDC) data from the 2015-16 school year survey were accessed (U.S. 
Department of Education Office for Civil Rights, 2016). Dual-enrollment or dual-credit programs are described 
in the CRDC school survey as providing “opportunities for high school students to take college-level courses 
offered by colleges, and earn concurrent credit toward a high school diploma and a college degree while still in 
high school,” and exclude Advanced Placement (AP) and International Baccalaureate (IB) programs. 
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Who is in the Sample? 


The seven P-TECH schools in this study opened on a rolling basis between 2011-2016. The sam- 
ple for the analyses includes ninth-grade cohorts of students who were offered admission to the 
schools between 2013-2017 — the years for which it was possible to identify admissions lotteries 
for the schools. In addition, lotteries are not included for any school in its first year of operation, 
because it is unclear whether students had full knowledge of the program they were applying to 
in each of the school’s inaugural years. 


Students in the study sample are followed for up to three follow-up years (first, second, 
and third year of high school). The staggered nature of the school openings — as well as the fact 
that the sample includes more recent cohorts — means that some students can only be followed 
for one or two years. Therefore, the sample of students with follow-up data gets smaller from one 
year to the next. The Year | sample includes a total of 3,161 students who were included in lot- 
teries for all seven P-TECH 9-14 schools between 2013-17; the Year 2 sample includes 2,164 
students who applied to all seven schools between 2013-16; and the Year 3 sample includes 1,203 
students who applied to six of the seven schools between 2013-15. To maximize the sample size 
for the analyses, the impact findings for any given follow-up year are based on all students whose 
outcomes could be measured that year.> However, to confirm that the pattern of effects across 
years is not confounded with changes in the student sample, impacts were also examined for the 
subsample of “stable” students in each year across all three follow-up years. These findings are 
presented in Appendix A. 


It is also worth noting that the students in the analysis sample are not evenly distributed 
across P-TECH 9-14 schools. Because some schools have been open longer than others, there are 
more cohorts from some schools than others. Only one school had lotteries in all five years that 
are analyzed, two schools had four years of lotteries, three schools had three years of lotteries, 
and one school had two years of lotteries. This means that the findings are more likely to represent 
the effect of the model for students who attended the subset of P-TECH schools with both more 
and larger lotteries. 


Characteristics of Students in the Study Sample 


Table 3.1 reveals that the population of students who apply to P-TECH 9-14 schools are 
predominantly Black and Hispanic, and come from families who live in census tracts where 
the median household income is approximately half a standard deviation below the average 
income level for the city as a whole. More than 70 percent of the sample had eighth-grade 
math and ELA scores that did not meet grade level standards. Additionally, approximately 
10 percent of the sample were classified as English Language Learners (ELLs) before high 


>For a description of the school lottery years and sample sizes, please see Appendix Table A.1. 
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Table 3.1 
Baseline Characteristics of P-TECH 9-14 Analytic Sample 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 
Characteristic Winners Members _ Difference Difference Difference 
Race/ethnicity (%) 
Hispanic 44.8 47.3 -2.6 -0.05 0.150 
Black 40.2 39.6 0.6 0.01 0.720 
White 4.7 3.9 0.8 0.05 0.271 
Asian 8.6 7.7 0.9 0.04 0.369 
Other 1.7 1.5 0.2 0.02 0.689 
Female (%) 37.7 42.2 -4.5 ** -0.09 0.013 
Median HH income of neighborhood® -0.5 -0.5 0.0 -0.01 0.849 
Missing median HH income (%) 2.6 2.4 0.2 0.01 0.807 
8th-grade ELA test performance level? 
Did not meet standards (level 1) (%) 26.1 24.5 1.6 0.04 0.339 
Partially met standards (level 2) (%) 44.8 43.7 1.0 0.02 0.618 
Fully met standards (level 3) (%) 21.6 23.0 -1.4 -0.03 0.385 
Met standards with distinction (level 4) (%) 4.1 4.5 -0.4 -0.02 0.598 
Missing test information (%) 3.5 4.3 -0.8 -0.04 0.315 
8th-grade math test performance level? 
Did not meet standards (level 1) (%) 36.9 36.9 0.1 0.00 0.977 
Partially met standards (level 2) (%) 35.1 35.2 -0.1 0.00 0.966 
Fully met standards (level 3) (%) 11.7 11.7 0.0 0.00 0.993 
Met standards with distinction (level 4) (%) 3.2 4.1 -0.8 -0.05 0.255 
Missing test information (%) 13.0 12.2 0.9 0.03 0.541 
Flagged as English Language Learner (%) 10.3 9.9 0.4 0.01 0.752 
Missing English Language Learner (%) 6.6 6.0 0.7 0.03 0.481 
Enrolled in a charter school in spring 
of grade 8 (%) 8.1 8.0 0.2 0.01 0.889 
Sample size (total=3,161) 1,479 1,682 


Number of lotteries (total=42) 
(continued) 


school.° Finally, students attended charter schools at similar levels across groups in the 
eighth-grade year, as seen in Table 3.1. 


°New York City began providing free lunch for all students during the 2017-18 school year, so all students 
are marked in data as receiving free meals in that year. Thus, for consistency across all years of data, instead of 
using eligibility for subsidized meals as an income proxy (a common measure of income in education research), 
the research team obtained data on median household income by census tract from the U.S. Census Bureau. 
Income levels were standardized across tracts, within students’ baseline eighth-grade year. Estimates are pre- 
sented in standard deviation units. 
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Table 3.1 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department 
of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as well as data from 
NYC DOE enrollment files from the 2012-2017 school years, and American Community Survey (ACS) data by 
census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a regression 
of a given baseline characteristic on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. 

The coefficient on the latter indicator variable equals the difference in the mean baseline characteristic 
for lottery winners and comparison group members. 

The value for comparison group members equals the corresponding value for P-TECH 9-14 lottery winners 
minus the estimated difference between lottery winners and comparison group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 

An F test was used to assess the statistical significance of the overall difference between lottery winners and 
control group members reflected by the full set of baseline characteristics in the table. The resulting p-value is not 
statistically significant (p-value = 0.625). 

aMedian household income has been z-scored to standardize across census tract and cohort year. 

>Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level for state math 
and reading exams. Due to missing test scores, the sum of levels 1-4 may not add to 100 percent. 


income of neighborhood, eighth-grade academic performance, or status as English language 
Leamers. Although there are statistically fewer female students in the program group, the size of 
the difference (4.5 percentage points) is relatively small.’ Importantly, an omnibus test also indi- 
cates that across all characteristics, there is no systemic difference between the program and com- 
parison groups. For these reasons, the groups are comparable, and differences in outcomes can 
be attributed to the P-TECH 9-14 program, and not to other factors or student characteristics. * 


While the baseline equivalency of the analytic sample is important for understanding 
whether the program and comparison groups are, in fact, comparable, it is also important to know 
whether the sample of students admitted to the P-TECH 9-14 schools via lottery was representa- 
tive of the overall population of students in the schools. A comparison of the analytic sample of 
program group students to the sample of students who were admitted to a P-TECH 9-14 school 
through a non-lottery process (that is, students who met the admissions criteria without oversub- 
scription) was also conducted. Results indicate that program group students in the analytic sample 
were more likely to have been classified as ELL students and had lower math and ELA test scores 
in eighth grade than students admitted to the P-TECH 9-14 schools without a lottery. They were 
also somewhat less likely to have attended a charter school in eighth grade, but were no more or 
less likely to be special education students.’ The implications of this are that the results of this 


"Baseline equivalency was also measured for the stable sample of students who are included in all three 
years of outcomes. A table of these results can be found in Appendix Table A.2. The observed gender imbalance 
is driven by several small lotteries in the early years and appear to be results of random bad draws for gender, 
since the HSAPS lottery does not consider student gender as a factor in the placement algorithm. 

8In addition, to account for this difference, the study team included gender in the regression model as a 
covariate. For more information on the model, please see Appendix A. 


Please see Appendix Table A.4 for results of this comparison. 


22 


study apply to the lowest-performing students who were offered seats in P-TECH 9-14 schools 
at baseline. Although the results may not be generalizable to the full sample of students enrolled 
in P-TECH 9-14 schools, they provide a policy-relevant set of analyses that address the impacts 
for students who may often be considered more academically precarious than their higher- 
performing peers. 


Understanding the Results 


The estimates presented in this report are what are known as intention-to-treat (ITT) estimates; 
that is, each group is made up of students who were assigned to either the program or comparison 
group by the lottery.'° All students who were assigned a seat in high school through a lottery for 
admission to a P-TECH 9-14 school were included in the sample. This does not mean that all 
students included in the program group actually enrolled in either the P-TECH 9-14 school to 
which they received an offer of admission, or the non-P-TECH 9-14 school to which they were 
assigned. The students in the sample are literally those for whom the assignments were “in- 
tended.” The reason for this is that the ITT estimate is the experimental estimate in which groups 
can be assumed to be comparable because, on average, they look similar and were similar in 
motivation in wanting to attend these schools; the only difference is whether they won or lost the 
lottery. In addition, this estimate is sometimes thought of as the policy-relevant estimate because 
it describes the effect of the intervention that policy makers can control; that is, policy makers can 
provide seats in particular kinds of schools, but they cannot force students to enroll in those 
schools. Thus, these results can be thought of as representing the best estimates of what can be 
expected, on average, to a community of students when a school or set of schools is opened and 
made available to them, rather than estimates of the average effects of student enrollment in the 
program.'! However, it is also worth noting that most students did comply with their intended 
group assignment (that is, most students assigned to P-TECH 9-14 schools enrolled in these 
schools), so the ITT estimates are roughly similar to the effects of enrolling in a P-TECH 9-14 
school. 


Impacts on School Choice 


In New York City, students participate in the high school admissions lottery during the 
winter of their eighth-grade year, with admissions offers usually sent out sometime during March 
of that year; students know what high school they have been admitted to prior to the end of middle 
school, and each student receives only one admission offer. Importantly, the lottery only includes 


'0Please see the Appendix A for the regression model equations. 

‘Tn order to understand the impacts for those students who actually enrolled in the schools to which they 
were assigned, the study team also calculated what are known as Complier Average Causal Estimates (CACE), 
or the estimates for those students who “complied” with their lottery assignment. However, because there was 
fairly high compliance with lottery placements in the study sample, these estimates do not differ greatly from the 
ITT estimates presented here. For information on compliance rates and the CACE estimates, please see Appendix 
Tables A.5 and A.6. 
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admissions for high schools run directly by NYC DOE and does not include public charter 
schools, which have their own admissions processes later in the spring. ! 


One unexpected outcome of the P-TECH 9-14 story is related to school choice. In partic- 
ular, there is a statistically significant difference between program and comparison group students 
in terms of the proportion of those who chose to enroll in a charter high school for ninth grade, 
rather than the NYC DOE school to which they were assigned. In particular, 9.5 percent of com- 
parison group students chose to enroll in a charter school rather than their assigned school, com- 
pared with only 5 percent of students who won the P-TECH 9-14 lottery, for a statistically signif- 
icant difference of 4.6 percentage points (see Table 3.2). Additionally, the difference in school 
enrollment choice primarily occurred for the ninth-grade year and persisted, with very few stu- 
dents leaving NYC DOE schools for charter schools in later years (or returning to the district from 
charter schools). These findings are despite the fact that students in the program and comparison 
groups were enrolled in charters at similar rates during the baseline eighth-grade year. This find- 
ing is interesting because it indicates that students assigned to P-TECH 9-14 schools initially 
appear to have been more satisfied with their school assignments than those assigned to other 
schools, as evidenced by the fact that they were less motivated to find alternative schooling after 
receiving their assignment to a P-TECH 9-14 school. 


Table 3.2 also shows the percentage of students who left the New York City public school 
system (marked as “inactive” in the system). The proportion of inactive students is similar in the 
program and comparison groups. While the number of students who exited the public system 
increases across follow-up years, the exit rate is similar across the two lottery assignment groups. 
This indicates that, unlike charter school enrollment, assignment to a P-TECH 9-14 school did 
not have an effect on student or family decisions to leave the public school system. 


The charter school finding has implications for some of the analyses. Namely, students 
who attended charter schools are missing more attendance and course enrollment outcome data 
than students enrolled in NYC DOE schools (Regents data are unaffected).'? More students in 
the comparison group attended charter schools, so a higher proportion of students in the com- 
parison group have missing attendance and credit data. By extension, this means that there 
could be a difference in the characteristics of students in the program and comparison groups 
with non-missing data. Additional analyses suggest that the two groups are similar in their ob- 
served characteristics at baseline.'* However, they could differ in unobserved ways, and these 
differences could be confounded with the effects of P-TECH 9-14 on attendance and credit 


By law, New York City charter schools must accept applications until at least April 1st. Admissions lot- 
teries for charter schools are then held after April 1, which is after NYC DOE HSAPS lottery admissions notifi- 
cations are sent to students. 

Regents data are collected and reported at the state level, while course and attendance level are collected 
and reported at the district level. Since charter schools do not report the same data to the district in the same way 
other schools do, these data appear not to have been reported to the district by most charter schools. 

4Analyses results are available upon request by emailing information@madrc.org. 
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Table 3.2 
Charter and Inactive Students for P-TECH 9-14 Analytic Sample 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 

Outcome Winners Members _ Difference Difference Difference 
Year 1 
Newly enrolled in a charter school Y1 (%) 4.9 9.5 -4.6 * -0.15 0.064 
Left the NYC public school system Y1 (%) 5.3 5.6 -0.3 -0.01 0.741 
Sample size (total=3,161) 1,479 1,682 
Number of lotteries (total=42) 
Year 2 
Newly enrolled in a charter school Y2 (%) 0.5 0.6 -0.1 -0.01 0.674 
Continuing enrollment in a charter 
school Y2 (%) 3.7 8.1 -4.5 ** -0.17 0.045 
Left the NYC public school system Y2 (%) 7.1 7.9 -0.9 -0.03 0.417 
Sample size (total=2,164) 1,090 1,074 
Number of lotteries (total=29) 
Year 3 
Newly enrolled in a charter school Y3 (%) 0.6 0.1 0.5 0.07 0.239 
Continuing enrollment in a charter 
school Y3 (%) 3.0 7.0 -4.0 * -0.16 0.094 
Left the NYC public school system Y3 (%) 8.5 11.4 -2.9 -0.09 0.119 
Sample size (total=1,203) 529 674 


Number of lotteries (total=15) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department 
of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as well as data from 
NYC DOE enrollment files from the 2012-2017 school years, and American Community Survey (ACS) data by 
census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a regression 
of a given outcome on a series of indicator variables that identify each lottery plus an indicator variable that 
equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable equals the 
difference in the mean outcome for lottery winners and comparison group members. The value for comparison 
group members equals the corresponding value for P-TECH 9-14 lottery winners minus the estimated difference 
between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test score, Missing 
rate of z-scored 8th-grade ELA test score, z-scored 8th-grade math test score, Missing rate of z-scored 8th-grade 
math test score). 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 
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accumulation. Appendix A discusses various sensitivity tests that were conducted to explore 
whether these missing data patterns are introducing bias into the results. Only those results for 
which there is confidence in the interpretation are discussed in the main text. 


Early Impacts on Credit Accumulation, Regents Exams, 
and Attendance 


This report primarily presents a first look at the impact of the P-TECH 9-14 model on students’ 
experiences after one, two, and three years of high school.!° The early impacts presented here 
demonstrate that P- TECH 9-14 students achieved early high school benchmarks: 


e P-TECH 9-14 students were more likely to earn CTE and other nonacademic 
course credits than their comparison school counterparts. 


e These extra credits translated into overall higher credit accumulation and did 
not appear to come at the expense of core academic credit accumulation. 


e More P-TECH 9-14 students also took and passed Regents exams with scores 
qualifying them for CUNY dual enrollment earlier in high school than students 
in the comparison group. 


Annual attendance, which is a common proxy for student engagement and success in 
school, was also measured. However, due to confounded issues with missing data patterns as 
described above, these estimates are less reliable than those presented for the other outcomes. 
The interim outcomes measured in this report provide important insights into whether key fea- 
tures of the P-TECH 9-14 experience might be expected to lead to later impacts in outcomes 
such as high school graduation, college enrollment, and degree attainment that the model was 
designed to improve. 


Academic and CTE Credits 


The credit outcomes examined include accumulation of academic high school credits 
necessary for students to move toward high school graduation, as well as CTE and other nonaca- 
demic credits that are emphasized as part of the P-TECH 9-14 model. 


Table 3.3 illustrates the impacts on the cumulative total credits earned; academic credits; 
and CTE and other nonacademic credits for students in P-TECH 9-14 schools!* This last measure 
counts all credit-bearing courses, including arts and health, and excluding math, English, science, 


'SThe study team measures outcomes for specific numbers of high school years after the lottery assignment, 
rather than grades. This is done to account for the fact that some students in the sample repeat grades, and a ninth- 
grade repeater, for example, would have two years of exposure to the program but still have the same grade 
designation as a new freshman. Thus, each of the years corresponds to years enrolled in high school, rather than 
high school grade designation. 


‘6In NYC DOE administrative records, 1 credit = 1 semester. 
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Table 3.3 
Credits Impacts for P-TECH 9-14 Analytic Sample 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 
Outcome Winners Members Difference Difference Difference 
Year 1 
Cumulative Total Credits Attempted Y1* 13.7 12.9 0.8 * 0.16 0.059 
Cumulative Total Credits Earned Y1* 11.7 11.2 0.5 0.10 0.113 
Cumulative Academic Credits Attempted Y1 9.6 10.0 -0.4 -0.11 0.424 
Cumulative Academic Credits Earned Y1 8.1 8.6 -0.5 -0.13 0.148 
Cumulative CTE/Other Credits Attempted Y1* 27 1.6 TA: e** 0.76 0.002 
Cumulative CTE/Other Credits Earned Y1* 2.5 1.4 i es ala 0.74 0.001 
Sample size (total=3, 161) 1,479 1,682 
Number of lotteries (total=42) 
Year 2 
Cumulative Total Credits Attempted Y2* 27.6 25.7 1.9 ** 0.21 0.031 
Cumulative Total Credits Earned Y2* 22.8 21.4 1.3 ** 0.14 0.039 
Cumulative Academic Credits Attempted Y2 19.6 20.2 -0.6 -0.08 0.419 
Cumulative Academic Credits Earned Y2 15.8 16.5 -0.8 -0.10 0.130 
Cumulative CTE/Other Credits Attempted Y2* 5.6 3.1 2.6 *** 1.00 <.0001 
Cumulative CTE/Other Credits Earned Y2* 5.0 2.7 230%" 0.94 <.0001 
Sample size (total=2, 164) 1,090 1,074 
Number of lotteries (total=29) 
Year 3 
Cumulative Total Credits Attempted Y3* 42.2 38.3 4.0 *** 0.27 0.004 
Cumulative Total Credits Earned Y3° 33.6 31.6 2.0 ** 0.13 0.023 
Cumulative Academic Credits Attempted Y3 30.0 29.6 0.4 0.04 0.733 
Cumulative Academic Credits Earned Y3 23.2 24.0 -0.8 -0.07 0.295 
Cumulative CTE/Other Credits Attempted Y3* 8.4 5.0 3.4 *** 0.87 <.0001 
Cumulative CTE/Other Credits Earned Y3* 7.3 4.4 2.8 *** 0.74 <.0001 
Sample size (total=1,203) 529 674 


Number of lotteries (total=15) 
(continued) 
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Table 3.3 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department of 
Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as well as data from NYC 
DOE enrollment files from the 2012-2017 school years, and American Community Survey (ACS) data by census tract 
for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the difference 
between P-TECH 9-14 lottery winners and comparison group members are obtained from a regression of a 

given outcome on a series of indicator variables that identify each lottery plus an indicator variable that equals 1 for 
lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable equals the difference in the mean 
outcome for lottery winners and comparison group members. The value for comparison group members equals the 
corresponding value for P-TECH 9-14 lottery winners minus the estimated difference between lottery winners and 
comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th Grade ELA test score, Missing rate 
of z-scored 8th Grade ELA test score, z-scored 8th-grade math test score, Missing rate of z-scored 8th-grade math 
test score). 

Sample sizes are from the ITT random assignment sample; each measure's data availability can be found in Table 
A.8. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 1 
percent; ** = 5 percent; * = 10 percent. 

Attempted credits measures include credits for all courses for which the student received either a passing or failing 
grade (so does not include courses awarded no grade). 

aCumulative Total Credits measures include all credits attempted or earned including Physical Education and 
Functional Code credits, while Cumulative CTE/Other Credits measures exclude Physical Education and Functional 
Code credits. 


social studies, foreign language, and physical education.'’ By the end of both the second and 
third years of high school, P-TECH 9-14 students earned more total credits than students in 
other schools, with results driven entirely by credit accumulation in CTE and other nonaca- 
demic subjects. By the end of three years in high school, P-TECH 9-14 students had earned an 
average of two total credits more than students in the comparison group, a finding being driven 
by students earning almost three more credits in CTE and other nonacademic courses. With an 
average of 34 total credits earned by P-TECH 9-14 students by the end of the third year, these 
students are on track to earn 10 credits in their final year of school to reach the 44 credits re- 
quired to graduate, while the students in the comparison group, who have earned an average of 
32 credits by the end of year three, will need to earn 12 credits in the fourth year of high school 
in order to graduate on time. Notably, the additional accumulation of CTE and other credits 
does not appear to come at the expense of academic credits earned during the same time periods. 
During each of the years examined, there were not statistically significant differences in the 
average number of core academic credits either attempted or earned by students in the program 
group and in the comparison group. 


Although nonacademic credits were earned in both CTE and non-CTE subjects, Figure 
3.1 illustrates the differences in CTE course types that students in each of the groups were 


"Because the CTE and other credit outcome does not include physical education credits, the estimates from 
the academic credits and CTE and other credits attempted outcomes do not sum to the estimates for total credits 
attempted outcomes. 
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Figure 3.1 
P-TECH 9-14 Analytic Sample: 


Program/Comparison Differences in % CTE/Other Courses Attempted 
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SOURCES: MDRC's calculations use High School Application Processing System and New York City Department of Education (NYC DOE) enrollment 
files from the 2013-2017 school years. 


engaged in. Not only did P-TECH 9-14 students earn more career-related course credits overall 
than the students in the comparison group, many of the kinds of courses the P-TECH 9-14 
students were enrolled in were different from the CTE courses taken by comparison group stu- 
dents. For example, P-TECH students earned far more credits in engineering, technology, 
work-based learning, and human services than students in the comparison group.'*® Additional 
analyses (see Appendix Table A.11) indicate that P-TECH 9-14 students made room for these 
kinds of courses in their schedules primarily by taking fewer arts, physical education, and health 
classes. While these subjects account for a relatively large portion of nonacademic credits 
earned by both groups of students, they were a much larger portion of total credits earned by 
comparison group students. For example, physical education and health comprised fully 51 
percent of the nonacademic credits earned by students in the comparison group, but just 41 
percent of credits earned by P-TECH 9-14 students. Likewise, while 21 percent of nonacademic 
credits earned by comparison group students were in the arts, just 15 percent were for P- TECH 
9-14 students. By contrast, 31 percent of nonacademic credits earned by P-TECH 9-14 students 
were in CTE fields, versus just 14 percent for comparison group students. 


Overall, not only were P-TECH 9-14 students earning more nonacademic credits than 
their comparison group counterparts, they were earning them in subjects aligned with the science, 
technology, engineering and mathematics (STEM) areas of focus associated with their schools’ 
themes, as well as in the work-based learning component of the model. By contrast, even when 
comparison group students did engage in CTE course work, the only area where they earned 
comparatively more credits than P-TECH 9-14 students was in courses in electrical work and 
other trades, which are more related to older models of vocational education than the newer mod- 
els of CTE that are showcased in the P-TECH 9-14 model.” 


Regents Exams 


Given that part of the P-TECH 9-14 model is an “early college” model that encourages 
dual enrollment in CUNY coursework, the study team examined whether students took New 
York State Regents exams more frequently, and whether they passed those exams with scores 
that qualified them for CUNY enrollment earlier in their high school careers and at higher levels 
than students in the comparison group.”° Table 3.4 illustrates the findings for Regents attempts 
and pass rates. In each of the first three years of high school, P-TECH 9-14 students attempted 
more Regents exams than students in comparison schools, and more of them passed the ELA 
Regents exam with a score qualifying them for enrollment in CUNY coursework than comparison 


'8The human services category largely applies to courses in health care careers. 
Rosen, Visher, and Beal (2018). 


?°In New York State, all public high school students must take and pass five Regents exams in order to 
graduate high school. These exams must be in English language arts, a math subject, science, social studies, and 
one other in a subject of choice. There is one pass score level required to graduate high school, but CUNY 
requires a higher passing score on ELA and math exams for students to be eligible to enroll in college-level 
course work. Any Regents exams taken prior to ninth grade were dropped from the analysis. 


30 


Outcome 


Year 1 
Cumulative Regents Attempted Y1 


Cumulative Regents Passed, score of 
65+ Y1 


Passed Math Regents to CUNY 
standard Y1 (%) 


Passed ELA Regents to CUNY 
standard Y1 (%) 
Sample size (total=3,161) 


Number of lotteries (total=42) 


Year 2 
Cumulative Regents Attempted Y2 


Cumulative Regents Passed, score of 
65+ Y2 


Passed Math Regents to CUNY 
standard Y2 (%) 


Passed ELA Regents to CUNY 
standard Y2 (%) 


Sample size (total=2,164) 
Number of lotteries (total=29) 


Year 3 
Cumulative Regents Attempted Y3 


Cumulative Regents Passed, score of 
65+ Y3 


Passed Math Regents to CUNY 
standard Y3 (%) 


Passed ELA Regents to CUNY 
standard Y3 (%) 


Sample size (total=1,203) 
Number of lotteries (total=15) 


Table 3.4 
Regents Exam Impacts for P-TECH 9-14 Analytic Sample 


P-TECH Comparison 


Lottery 
Winners 
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Group 
Members 


1.9 


1.1 


31.5 


1.4 


1,682 


4.8 


2.7 


40.0 


25.2 


1,074 


7.7 


4.0 


40.8 


50.6 


674 


Estimated 
Difference 


0.7 RK 


0.2 
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6.8 * 
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oa 


2.6 


16.8 *** 


1.9 KK 


0.4 ** 


6.8 
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Effect Size of P-Value for 


Estimated 
Difference 


0.50 


0.16 


0.07 


0.32 


0.54 


0.18 


0.05 


0.42 


0.40 


0.14 


0.14 


0.16 


Estimated 
Difference 


0.003 


0.142 


0.408 


0.051 


<.0001 


0.005 


0.448 


<.0001 


0.000 


0.040 


0.137 


0.006 


(continued) 


Table 3.4 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as 
well as data from NYC DOE enrollment files from the 2012-2017 school years, and American Community 
Survey (ACS) data by census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a 
regression of a given outcome on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable 
equals the difference in the mean outcome for lottery winners and comparison group members. The value for 
comparison group members equals the corresponding value for P-TECH 9-14 lottery winners minus the 
estimated difference between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test score, 
Missing rate of z-scored 8th Grade ELA test score, z-scored 8th-grade math test score, Missing rate of z- 
scored 8th-grade math test score). 

Sample sizes are from the ITT random assignment sample; each measure's data availability can be found in 
Table A.8. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Regents attempts variables only include Regents exams for which the student received a grade and does 
not include exams where the student was absent, ill, cheated, or the school misadministered the exam. 


students in the same time period. At the end of two years of high school, 42 percent of P- TECH 
9-14 students had passed the ELA Regents with a CUNY-qualifying score, compared with 25 
percent of comparison group students. By the end of the third year, the gap was smaller but still 
favored P-TECH 9-14 students.”! Overall, P-TECH 9-14 students did pass more Regents exams 
than other students, although this appears to be a function of attempting the tests more frequently. 
Beginning in the first year of high school, 87 percent of P-TECH 9-14 students attempted at least 
one Regents exam, compared with 79 percent of comparison group students.?? However, the 
greater level of attempts may be due to students retaking exams after passing at the minimum 65 
level, in an effort to reach the higher CUNY benchmarks and qualify for dual enrollment. On 
average, 27 percent of P-TECH 9-14 students retook a math or ELA Regents exam after receiving 
a 65, compared with 11 percent of comparison students. 


Because of these extra attempts, on average, P-TECH 9-14 students passed a smaller 
proportion of the exams they took. Specifically, they passed 47 percent of exams attempted by 
the end of the second year, compared with an average of 56 percent of exams taken by comparison 
group students in the same time period.”* It’s worth noting that despite the STEM focus in the 
P-TECH 9-14 schools, there are not statistically significant differences in pass rates between 
P-TECH students and the comparison group on math Regents exams at CUNY-qualifying levels. 
To be sure, students in this sample had very low baseline math achievement, which may make 
early attainment of college level math readiness a particular stretch. That said, by the end of the 


>! Any Regents exams taken prior to ninth grade were dropped from the outcome, as these attempts occurred 
preprogram. 

2? The calculation is available upon request by emailing information@mdrc.org 

23 Figures are derived from Table 3.4, where the average of exams passed at the 65 level is divided by the 
average of exams attempted for each group. 
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third year, P-TECH 9-14 students had passed slightly more math Regents exams at the base level 
score of 65 needed for high school graduation than students in the comparison sample (see Ap- 
pendix Table A.10) 


The ELA Regents tell a different story. According to CUNY staff, most students find 
passing the ELA tests easier; the pattern of higher ELA pass rates in this sample is similar to that 
of pass rates for incoming CUNY students overall. These findings indicate that statistically sig- 
nificantly more P-TECH 9-14 students are more prepared to enroll in college-level course work 
in ELA even by the end of one year of high school, and that difference increased by the end of 
two years of high school, as seen in Table 3.4. This is important because it indicates that at the 
end of two years of high school, far more P-TECH 9-14 students were eligible to dual enroll at 
CUNY than students in the comparison group during the same time period. 


Attendance 


During the first year of high school, P-TECH 9-14 students do not appear to have attended 
school more days than students in other schools (see Table 3.5). Although the main impact esti- 
mate does appear statistically significant, the result does not hold up to the sensitivity tests that 
were conducted to explore whether the findings could be confounded by differences in missing 
data across the program and comparison group (see Appendix A). However, by the end of the 
second year of high school, P-TECH 9-14 students were statistically significantly more likely to 
attend high school for more days. In a 180-day school year, P-TECH 9-14 students on average 
attended three to five more days of school during the second year.”> By the end of the third year 
there is no longer a robust, statistically significant difference between groups on attendance. 


>4Policies related to eligibility shifted over the time periods included in this study, which may impact student 
overall eligibility for enrollment in CUNY courses in math. For example, in 2015, CUNY required both a passing 
Regents score and completion of algebra 2 or trigonometry. Students could also have earned eligibility with a 
500 or higher math SAT score, a 21 or higher math ACT score, or another CUNY math assessment. In 2017, the 
policy changed to just include exam passing scores, but not course completion. 

>>The estimates are 1.8 to 3.1 more of a school year. The report team translated this into days by taking 1.8 
percent of a 180-day school year, which is 3.24 days at the low end; and 3.1 percent of 180 school days, or 5.58 
days at the high end. 
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Table 3.5 
Attendance Impacts for P-TECH 9-14 Analytic Sample 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated — Estimated 

Outcome Winners Members Difference Difference _ Difference 
Year 1 
Attendance (180 days) Y1 (%) 78.5 76.6 19* fT 0.08 0.082 
Sample size (total=3,161) 1,479 1,682 
Number of lotteries (total=42) 
Year 2 
Attendance (180 days) Y2 (%) 74.3 71.2 3:17" 0.11 0.004 
Sample size (total=2,164) 1,090 1,074 
Number of lotteries (total=29) 
Year 3 
Attendance (180 days) Y3 (%) 70.6 66.8 3.8** TF 0.12 0.037 
Sample size (total=1,203) 529 674 


Number of lotteries (total=15) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as 
well as data from NYC DOE enrollment files from the 2012-2017 school years, and American Community 
Survey (ACS) data by census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners.Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a 
regression of a given outcome on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter 

indicator variable equals the difference in the mean outcome for lottery winners and comparison group 
members. The value for comparison group members equals the corresponding value for P-TECH 9-14 lottery 
winners minus the estimated difference between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test score, 
Missing rate of z-scored 8th-grade ELA test score, z-scored 8th-grade math test score, Missing rate of z- 
scored 8th-grade math test score). 

Sample sizes are from the ITT random assignment sample; each measure's data availability can be found 
in Table A.8. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: 
*** = 1 percent; ** = 5 percent; * = 10 percent. 

+ This result is not robust to the upper-bound test in the bounding exercise. See Technical Appendix for 
further discussion. 
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Chapter 4 


Discussion and Next Steps 


This report provides an early look at the implementation and interim impacts from the evaluation 
of the New York City P-TECH 9-14 schools, the first rigorous study of the model. It provides 
background on the development of the model and describes its high school, college, and work- 
based learning (WBL) components. The report also begins to describe the variations in imple- 
mentation across the seven schools in the study. The impact findings suggest that students in these 
schools are meeting the benchmarks the schools are designed to help them achieve. The findings 
may provide insight into what could predict future successes for these students, many of whom 
entered ninth grade with weak academic achievement in both English language arts (ELA) and 
math. 


The seven P-TECH 9-14 schools in the report have several features in common, including 
accelerated schedules, opportunities for college coursework, and WBL activities in an integrated 
six-year scope and sequence. At the same time, based on interviews with school staff members, 
it is clear that there is variation across schools in the implementation of the P-TECH 9-14 model. 
Beyond focusing on different career and technical education (CTE) subjects and college majors, 
the schools have different ways of supporting and advising their students, 


The schools also have internal variations. Two students in the same school may have very 
different experiences, depending on the extent and timing of their college courses and their par- 
ticipation in WBL activities. School requirements may also be obstacles to opportunity. Some are 
structural and set by institutions outside of the P-TECH 9-14 school, such as the need for students 
to pass Regents exams in order to take college courses, or to have proper paperwork to qualify 
for a paid internship. Other requirements are more arbitrary and set by the individual high school. 
For example, students may need a letter of recommendation to take college classes, or they may 
be required to complete an extensive internship application process. Many of these requirements 
are in place to ensure that students have the greatest chance of succeeding when they enter the 
high-stakes environment of an internship or college. However, the requirements also have equity 
implications, as this means that individual students may have different access to opportunities. 
The final study report will explore these variations in detail, along with the facilitators and barriers 
to implementation of the many elements of a P-TECH 9-14 school. It will also examine the dif- 
ferences between P-TECH 9-14 schools and those attended by comparison group students. 


The results of this interim report present encouraging early findings about the ways 
P-TECH 9-14 schools are affecting students’ high school experiences and the potential transition 
to college and career settings. 


In particular, the increased accumulation of CTE and other nonacademic credits is evi- 
dence that these schools are providing students with greater levels of career-related exposure than 
students enrolled in other schools. Furthermore, the variety of CTE courses in P-TECH 9-14 
schools suggests that these schools are providing different and potentially more modern career 
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experiences. Additionally, the schools are helping students succeed in CTE in ways that do not 
appear to be at the expense of earning academic credits. The tradeoff that students are making 
appears to be that they are earning fewer credits in the arts, physical education, and health than 
their counterparts in other schools, indicating that P-TECH 9-14 students are using their time in 
high school to accumulate credits in fields related to their target careers. 


Positive findings on students passing ELA Regents exams sooner and at CUNY- 
qualifying levels indicate that the P-TECH 9-14 model is successfully preparing them to take 
advantage of dual-enrollment opportunities at earlier points in high school than their counterparts 
in other schools. In addition, P-TECH 9-14 students ended their third year of high school with 
more Regents passed at the high school graduation level of 65. They also had higher overall credit 
accumulation, with fewer credits left to earn during the fourth year of high school. This indicates 
that more P-TECH 9-14 students may be prepared to reach high school graduation than students 
in the comparison group. 


Importantly, these successes are found for a sample representing the lowest- performing 
students enrolled in their schools, including those that entered with the lowest levels of academic 
success in eighth grade. The findings indicate that the P-TECH 9-14 school model appears to be 
meeting the needs of academically low-performing students. 


Although this report provides encouraging early evidence about the ability of the 
P-TECH 9-14 schools to prepare students for the transition to college and career, these are not 
definitive findings about the overall efficacy of the P-TECH 9-14 model or the success of the 
students who attend these schools. For example, as of this publication, two of the ninth-grade 
cohorts included in the analysis sample have yet to complete high school. So while it is encour- 
aging to learn that P-TECH 9-14 students are outperforming their peers on several of the measures 
most closely tied to the P-TECH 9-14 experience, it is not yet clear how these outcomes will 
affect dual-enrollment take-up or success, high school graduation, or college degree completion. 
These questions will be explored in subsequent reports. 


From a policy perspective, the findings do highlight ways in which a different kind of 
high school design can have a positive impact on important secondary school milestones, partic- 
ularly as they relate to the potential for smoother transitions to college and careers. The final 
report will also include a cost analysis that estimates how the cost of these schools compares to 
other programs that may or may not achieve the same results. 


Finally, these results should be viewed within the New York City context, which may 
have implications for the ability to generalize the findings. For example, New York City’s rich 
and varied labor market, as well as its robust public transit system, may give P-TECH 9-14 stu- 
dents and schools access to a larger variety of employer partners and WBL opportunities and the 
means to take advantage of them than students in other locations. 


Future impact and implementation research planned for this study will present a more in- 
depth comparison of outcome measures across schools, as well as a deeper look at the kinds of 
school settings experienced by the comparison students. The cost study will provide practical 
perspective on how these schools use resources toward achieving their desired outcomes. 
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Appendix A 


Additional Information About the Study 


More Information About the NYC Public School Lottery 


MDRC has a long history of conducting rigorous studies using data from the New York City high 
school admissions lottery, most notably through an ongoing study of students admitted to the 
city’s “small schools of choice,” which have nonselective admissions and serve many disadvan- 
taged young people. Much of the background for this report relies on the understanding of the 
mechanics of the lottery system that was described and written about in those earlier studies as 


well as work done by the creators of the New York City high school admissions algorithm. ! 


Students may be entered into a lottery by the admissions algorithm in several ways. For 
example, if a student lists a P- TECH 9-14 school as their first choice and they meet an admis- 
sions category for which they are eligible, that student will be placed into the lottery for the 
given school that they ranked first if there are more students who meet those criteria than there 
are available seats. 


However, students may also be placed into a lottery if they ranked a P-TECH 9-14 school 
lower down on their choice list. This can happen in several ways: (1) The student may have been 
placed into lotteries for higher-ranked schools and lost those lotteries; (2) the student did not com- 
pete in any lotteries for the higher-ranked schools because, for example, they were ineligible for 
admission based on the school’s preference categories, such as meeting the GPA or attendance 
criteria; (3) all the schools ranked higher on the student’s list were full by the time the algorithm 
reached that particular student. Like other researchers who have written extensively about using 
the New York City high school admissions lottery for rigorous research purposes, the study team 
included in the analytic sample for this study only students for whom the P-TECH 9-14 lottery 
was the first one they were placed into and who did not compete in any other lotteries prior to that 
one. This is because in order for the program and comparison students to be truly comparable in 
a way that makes the random assignment valid between groups, all students in both groups needed 
to have the same probability of selection for the analytic sample. Students who competed in earlier 
lotteries for higher-ranked schools would have different probabilities of being placed into the 
lotteries of interest for this study, dependent on their chances of admission in the previous lottery. 
Each prior lottery changes the probability of being placed in the P-TECH 9-14 lottery sample, 
and thus changes the random nature of the lottery sample. For this reason, students who were 
placed in other lotteries prior to being placed into the P-TECH 9-14 lottery were not included in 
the analytic sample. 


Sample Exclusions 


To create the analytic sample, several categories of students were dropped from the data. Specif- 
ically, students who applied to the lottery from private schools in eighth grade were not included, 
and students who were placed into lotteries with fewer than five students were also dropped from 
the analysis. This was done because overly small lotteries are more likely to be extremely 


‘Abdulkadiroglu, Agarwal, and Pathak (2015); Abdulkadiroglu, Angrist, Narita, and Pathak (2017); Bloom 
and Unterman (2013); Bloom and Unterman (2014). 
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unbalanced in terms of the number of students who end up in each assignment condition. Drop- 
ping the small lotteries provided assurance that the same lotteries would remain in the analyses 
and not drop out due to differences in missing outcome data within lotteries. Other students 
dropped from the sample include those who opted out of the lottery process and those who were 
ineligible for programs they applied for. 


Baseline Equivalence Between the Program 
and Comparison Group 


Based on the slight gender imbalance that was observed across program and comparison groups 
in the baseline equivalence table, the data were investigated and it appears that this statistically 
significant difference was primarily driven by several of the special education (SPED) lotteries 
and one general education lottery that have significant imbalances in male and female admissions, 
high enough to matter for the overall gender balance of the sample. 


When these lotteries were dropped from baseline equivalency analyses, the p-value on 
the gender difference became nonsignificant. While there does appear to be an imbalance in ac- 
ceptance rates along gender lines in these lotteries, the imbalance is likely due to random chance 
as a function of the New York City matching algorithm (which does not consider gender in stu- 
dent assignment), rather than intentional bias. The lotteries in question had gender applicants and 
acceptances as follows: 


Type Female Male % of Female | % of Male 
Applicants Applicants Applicants Applicants 
Accepted Accepted 
Gen Ed 52 149 7.7% 16.1% 
SPED 37 283 8.1% 27.6% 


School Cohorts, Sample Sizes, and Stable Sample Analyses 


Appendix Table A.1 provides a visual representation of the lottery cohorts by year that were in- 
cluded in the analytic sample. Each cell provides the sample size for each school and year lottery. 


P-TECH 9-14 Schools and Cohorts Included in the Analytic Samples 


Because of the shifting nature of these cohorts, baseline equivalency analysis on the 
stable sample was also conducted. The stable sample is defined as the subsample of students 
that can be followed across all three years of outcomes. This sample is identical to the sample 
of students included in the Year 3 analysis in the main report. This was done to provide addi- 
tional assurance that this subsample was balanced, and that the baseline equivalency was not 
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Appendix Table A.1 
P-TECH 9-14 Schools and Cohorts Included in the Analytic Samples 


Analytic Cohorts 
9th- Grade] School | School | School | School | School | School | School | Total Year 1 Year 2 Year 3 
Year 1 2 3 4 5 6 7 Results Results Results 
2013-14 13 13 
2014-15 194 130 154 478 
2015-16 268 179 36 99 121 9 712 


2016-17 183 169 117 117 112 45 218 961 


2017-18 248 141 121 112 171 21 183 997 


Total 


906 619 428 328 404 75 401 3161 


found only within the sample of students for whom only one year of outcome data were avail- 
able, as seen in Appendix Table A.2. 


Within this subsample, the same imbalance in female participation is found as in the 
overall sample, but this is not surprising given that the imbalance was found in a few early year 
lotteries, as described in the main text. Again, an omnibus test was not significant, providing 
confidence that the sample is balanced over time. 


In addition, to ensure that the impacts on outcomes presented for the Year 1 and Year 2 
samples were not driven entirely by impacts from the subsample of students that is stable across 
all years, impact analyses for Year 1 and Year 2 outcomes were also conducted using the stable 
sample, as seen in Appendix Table A.3. 


Within Year 1 for the stable sample, there are statistically significant differences on both 
Regents attempts and pass rates at the base pass level, as well as at levels qualifying students for 
CUNY in both math and English language arts (ELA). There are also statistically significant im- 
pacts for both taking and passing career and technical education (CTE) credits. 


In Year 2 for this sample, there are statistically significant impacts on all Regents out- 
comes except passing the math Regents with a CUNY-qualifying score. In Year 2 there are also 
statistically significant differences in accumulation of CTE credits. Finally, in Year 2 there is also 
an observed impact for attendance, in which P-TECH 9-14 students have an average attendance 
rate of 75.4 percent, compared to 71 percent for comparison group students. 


The results of the stable sample analysis are similar to those presented for the full sample 
for each of the individual year cohorts, indicating that the results are not unique to the stable 
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Appendix Table A.2 


Baseline Characteristics of P-TECH 9-14 Analytic Sample: 
Stable Sample (Cohorts 2013-2015) 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 
Characteristic Winners Members _ Difference Difference Difference 
Race/ethnicity (%) 
Hispanic 46.3 47.0 -0.6 -0.01 0.826 
Black 41.8 41.7 0.1 0.00 0.985 
White 3.4 2.6 0.8 0.05 0.443 
Asian 7.0 6.5 0.5 0.02 0.706 
Other 1.5 2.3 -0.8 -0.06 0.407 
Female (%) 41.8 50.2 -8.4 *** -0.17 0.005 
Median HH income of neighborhood* -0.5 -0.5 0.0 -0.06 0.354 
Missing median HH income (%) 0.6 1.2 -0.7 -0.06 0.409 
8th-grade ELA test performance level? 
Did not meet standards (level 1) (%) 34.2 30.9 3.3 0.07 0.266 
Partially met standards (level 2) (%) 50.1 49.2 0.9 0.02 0.784 
Fully met standards (level 3) (%) 12.3 14.9 -2.6 -0.07 0.271 
Met standards with distinction (level 4) (%) 1.9 1.2 0.7 0.06 0.371 
Missing test information (%) 1.5 3.9 -2.4 ** -0.15 0.027 
8th-grade Math test performance level? 
Did not meet standards (level 1) (%) 39.9 42.2 -2.3 -0.05 0.462 
Partially met standards (level 2) (%) 38.4 33.6 48 0.10 0.135 
Fully met standards (level 3) (%) 9.6 11.6 -2.0 -0.07 0.330 
Met standards with distinction (level 4) (%) 1.7 1.9 -0.2 -0.01 0.845 
Missing test information (%) 10.4 10.6 -0.2 -0.01 0.909 
Flagged as English Language Learner (%) 10.6 10.7 -0.1 0.00 0.966 
Missing English Language Learner (%) 0.2 0.1 0.1 0.02 0.744 
Enrolled in a charter school in spring 
of Grade 8 (%) 4.0 3.7 0.3 0.01 0.860 
Sample size (total=1,203) 529 674 


Number of lotteries (total=15) 
(continued) 
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Appendix Table A.2 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department 
of Education (NYC DOE) state test data for eighth-graders from the 2012-2014 school years, as well as data from 
NYC DOE enrollment files from the 2012-2015 school years, and American Community Survey (ACS) data by 
census tract for median household income from the 2012-2015 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a regression 
of a given baseline characteristic on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable 
equals the difference in the mean baseline characteristic for lottery winners and comparison group members. The 
value for comparison group members equals the corresponding value for P-TECH 9-14 lottery winners minus the 
estimated difference between lottery winners and comparison group members. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 

An F test was used to assess the statistical significance of the overall difference between lottery winners and 
control group members reflected by the full set of baseline characteristics in the table. The resulting p-value is not 
statistically significant (p-value = 0.206). 

aMedian household income has been z-scored to standardize across census tract and cohort year. 

>Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level for state 
math and reading exams. Due to missing test scores, the sum of levels 1-4 may not add to 100 percent. 


sample and are not confounded with the inclusion of additional cohort members in each of the 
years of outcomes. The exception to this is the stable sample result in Year 1 for passing the math 
Regents exam to the CUNY standard. This finding indicates that CUNY-standard pass rates in 
math appear to have been slightly stronger for earlier cohorts of students than for later cohorts. 


Students in the Study Sample Compared to All Enrolled Students 


Tests were also conducted comparing the sample of students who enrolled in P-TECH 9-14 
schools after winning the lottery to the sample of students who received admissions offers to 
P-TECH 9-14 schools either through a noncompetitive process or through a lottery that was not 
the first one they competed in.? 


Appendix Table A.4 indicates that students who enrolled in P-TECH 9-14 after receiving 
a placement via lottery look somewhat different from students who enrolled in P-TECH 9-14 
either not having been placed by lottery or having been placed through a later lottery. Specifically, 
these students performed significantly lower in both math and ELA in eighth grade. They were 
more likely to have received the lowest score of “Did Not Meet Standards” on both exams in 
eighth grade than other students enrolled in the schools. They were more likely to have been an 
English language learner (ELL) student, and less likely to have been enrolled in a charter school 
prior to ninth grade. 


Note that only 209 students were placed via a lottery that was not their first-choice lottery. The other 1,114 
students in this sample received placements through a noncompetitive process. 
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Outcome 


Year 1 
Attendance (180 days) Y1 (%) 
Cumulative Total Credits Attempted Y1° 


Cumulative Total Credits Earned Y1° 


Cumulative Academic Credits 
Attempted Y1 


Cumulative Academic Credits 
Earned Y1 


Cumulative CTE/Other Credits 
Attempted Y1° 


Cumulative CTE/Other Credits 
Earned Y1° 


Cumulative Regents Attempted Y1 


Cumulative Regents Passed, score of 
65+ Y1 


Passed Math Regents to CUNY 
standard Y1 (%) 


Passed ELA Regents to CUNY 
standard Y1 (%) 


Newly enrolled in a charter 
school Y1 (%) 


Left the NYC public school 
system Y1 (%) 


Sample size (total=1,203) 
Number of lotteries (total=15) 


Appendix Table A.3 
Impacts for P-TECH 9-14 Analytic Sample: Stable Sample (Cohorts 2013-2015) 
Effect Size of P-Value for 


P-TECH Comparison 


Lottery 
Winners 


79.8 
14.3 
12.0 


9.9 


8.0 


3.2 


2.9 
3.0 


1.4 


24.2 


12.1 


3.4 


4.3 
529 


Group 


Members 


44 


77.8 
13.0 
11.3 


10.1 


8.7 


1.6 


1.4 
1.8 


1.0 


17.3 


674 


Estimated 
Difference 


2.0 


6.9 * 


12.6 ** 


-4.8 * 


-1.1 


Estimated 
Difference 


0.08 
0.27 
0.15 


-0.06 


-0.16 


0.97 


0.95 
0.92 


0.39 


0.16 


0.58 


-0.18 


-0.05 


Estimated 
Difference 


0.200 
0.015 
0.044 


0.745 


0.159 


<.0001 


<.0001 
0.003 


0.029 


0.076 


0.011 


0.084 


0.379 


(continued) 


Appendix Table A.3 (continued) 


P-TECH Comparison 


Lottery 
Outcome Winners 
Year 2 
Attendance (180 days) Y2 (%) 75.4 
Cumulative Total Credits Attempted Y2* 28.8 
Cumulative Total Credits Earned Y2* 23.6 
Cumulative Academic Credits 
Attempted Y2 20.4 
Cumulative Academic Credits 
Earned Y2 16.2 
Cumulative CTE/Other Credits 
Attempted Y2? 6.0 
Cumulative CTE/Other Credits 
Earned Y2? 5.3 
Cumulative Regents Attempted Y2 6.8 
Cumulative Regents Passed, score of 
65+ Y2 3.1 
Passed Math Regents to CUNY 
standard Y2 (%) 37.1 
Passed ELA Regents to CUNY 
standard Y2 (%) 45.0 
Newly enrolled in a charter 
school Y2 (%) 0.2 
Continuing enrollment in a charter 
school Y2 (%) 3.0 
Left the NYC public school 
system Y2 (%) 6.4 
Sample size (total=1,203) 529 


Number of lotteries (total=15) 
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Group 
Members 


71.0 
26.0 
21.6 


20.3 


16.6 


3.2 


2.8 
4.8 


2.5 


31.4 


23.3 


0.5 


7.8 


8.7 


674 


Estimated 
Difference 


ape 
2.9 RK 
2.0 KKK 


0.1 


5.6 


21.7 *** 


-0.4 


-4.7 * 


Effect Size of P-Value for 


Estimated 
Difference 


0.15 
0.30 
0.20 


0.01 


-0.04 


1.06 


0.99 
0.70 


0.29 


0.12 


0.54 


-0.05 


-0.18 


-0.08 


Estimated 
Difference 


0.003 
0.004 
0.005 


0.920 


0.605 


<.0001 


<.0001 
<.0001 


0.002 


0.238 


<.0001 


0.314 


0.054 


0.138 


(continued) 


Appendix Table A.3 (continued) 


Outcome 
Year 3 
Attendance (180 days) Y3 (%) 


Cumulative Total Credits Attempted Y3" 


Cumulative Total Credits Earned Y3" 


Cumulative Academic Credits 
Attempted Y3 


Cumulative Academic Credits 
Earned Y3 


Cumulative CTE/Other Credits 
Attempted Y3° 


Cumulative CTE/Other Credits 
Earned Y3° 


Cumulative Regents Attempted Y3 


Cumulative Regents Passed, score of 
65+ Y3 


Passed Math Regents to CUNY 
standard Y3 (%) 


Passed ELA Regents to CUNY 
standard Y3 (%) 


Newly enrolled in a charter 
school Y3 (%) 


Continuing enrollment in a charter 
school Y3 (%) 


Left the NYC public school 
system Y3 (%) 


Sample size (total=1,203) 
Number of lotteries (total=15) 


Winners 


70.6 
42.2 
33.6 


30.0 


23.2 


8.4 


7.3 
9.7 


44 


47.6 


58.4 


0.6 


3.0 


8.5 


529 
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P-TECH Comparison 
Lottery 


Effect Size of P-Value for 


Group Estimated Estimated 
Members _ Difference Difference 
66.8 3.8%** 0.12 
38.3 4.0 *** 0.27 
31.6 2.0 ** 0.13 
29.6 0.4 0.04 
24.0 -0.8 -0.07 
5.0 3.4 *** 0.87 
4.4 2.8 *** 0.74 
7.7 1.9 *** 0.40 
4.0 0.4 ** 0.14 
40.8 6.8 0.14 
50.6 7.8 *** 0.16 
0.1 0.5 0.07 
7.0 -4.0 * -0.16 
11.4 -2.9 -0.09 
674 


Estimated 
Difference 


0.037 
0.004 
0.023 


0.733 


0.295 


<.0001 


<.0001 
0.000 


0.040 


0.137 


0.006 


0.239 


0.094 


0.119 


(continued) 


Appendix Table A.3 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department 
of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as well as data from 
NYC DOE enrollment files from the 2012-2017 school years, and American Community Survey (ACS) data by 
census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a regression 
of a given outcome on a series of indicator variables that identify each lottery plus an indicator variable that 
equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable equals the 
difference in the mean outcome for lottery winners and comparison group members. The value for comparison 
group members equals the corresponding value for P-TECH 9-14 lottery winners minus the estimated difference 
between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test score, Missing 
rate of z-scored 8th-grade ELA test score, z-scored 8th-grade math test score, Missing rate of z-scored 8th-grade 
math test score). 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes are from the ITT random assignment sample; each measure's data availability can be found in 
Table A.8. 

Attempted credits measures include credits for all courses for which the student received either a passing or 
failing grade (so does not include courses awarded no grade). 

Regents attempts variables only include Regents exams for which the student received a grade and does not 
include exams where the student was absent, ill, cheated, or the school misadministered the exam. 

aCumulative Total Credits measures include all credits attempted or earned including Physical Education and 
Functional Code credits, while Cumulative CTE/Other Credits measures exclude Physical Education and 
Functional Code credits. 

t This result is not robust to the upper-bound test in the bounding exercise. See Technical Appendix for further 
discussion. 


These differences mean that the results of this study should be interpreted with some 
caution and are only valid for the population of students included in the analytic sample, not for 
the population of students enrolled in P-TECH 9-14 schools overall. 


The Model 


The same impact model was used for all outcomes described in the report. The primary equation 
for doing so was: 


Y; = arty Ay + Bo * Tj + Oy * Sui + Op * Spi + OG * Sei + & (1) 


where Y; is a relevant outcome for student 7; Tis a lottery winner indicator equal to 1 if 
student i wins lottery 7 and 0 otherwise; / is a vector of lottery indicators equal to 1 for lottery 7 
and 0 otherwise;* Sy; and Sp; are student 7’s eighth-grade scores on New York State tests of math 
and reading (included for precision); Sg; is an indicator for student gender and ¢; is a random 


3Lotteries with fewer than five lottery winners or lottery losers were dropped from the sample. In addition, 
because each school reserves a certain number of seats for special education students, students with special edu- 
cation status are admitted through separate lotteries. For each school and year, lotteries for both general and 
special education were identified separately, and both were included in the model. 
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Appendix Table A.4 


Baseline Characteristics of P-TECH 9-14 Analytic Sample 


Who Competed in a First 


Characteristic 


Race/ethnicity (%) 
Hispanic 
Black 
White 
Asian 
Other 


Female (%) 


Median HH income of neighborhood? 
Missing median HH income (%) 


8th-grade ELA test performance level® 
Did not meet standards (level 1) (%) 


Partially met standards (level 2) (%) 

Fully met standards (level 3) (%) 

Met standards with distinction (level 4) (%) 
Missing test information (%) 


8th-grade math test performance level® 
Did not meet standards (level 1) (%) 


Partially met standards (level 2) (%) 

Fully met standards (level 3) (%) 

Met standards with distinction (level 4) (%) 
Missing test information (%) 


Flagged as English Language Learner (%) 
Missing English Language Learner (%) 


Flagged as Special Education (%) 


Enrolled in a charter school in spring 
of Grade 8 (%) 


Sample size (total=2,802) 
Number of cohorts (total=24) 


Lottery vs. Other P-TECH 9-14 Offer Students 


P-TECH Other Effect Size of P-Value for 
Lottery P-TECH Estimated Estimated Estimated 
Winners Offerees* Difference Difference Difference 
44.8 41.8 2.9 0.06 0.153 
40.2 42.8 -2.6 -0.05 0.184 
4.7 4.5 0.2 0.01 0.836 
8.6 9.7 -1.1 -0.04 0.383 
1.7 1.1 0.6 0.04 0.341 
37.7 39.4 -1.7 -0.03 0.412 
-0.5 -0.4 0.0 -0.06 0.131 
2.6 3.4 -0.8 -0.05 0.283 
26.1 16.1 10.0 *** 0.24 <.0001 
44.8 44.7 0.0 0.00 0.994 
21.6 28.5 -7.0 *** -0.16 0.000 
4.1 6.1 -2.1 ** -0.10 0.038 
3.5 4.5 -0.9 -0.05 0.282 
36.9 24.9 12.0 *** 0.25 <.0001 
35.1 38.8 -3.7 * -0.08 0.094 
11.7 15.4 -3.7 ** -0.11 0.015 
3.2 4.0 -0.7 -0.04 0.392 
13.0 16.8 -3.8 ** -0.11 0.020 
10.3 7.5 2.8 ** 0.09 0.040 
6.6 8.4 -1.8 -0.07 0.121 
18.2 17.3 0.9 0.02 0.613 
8.1 11.7 -3.6 *** -0.12 0.008 
1,479 1,323 


(continued) 
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Appendix Table A.4 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department 
of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as well as data from 
NYC DOE enrollment files from the 2012-2017 school years, and American Community Survey (ACS) data by 
census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and other P-TECH 9-14 offerees are obtained from a regression 
of a given baseline characteristic on a series of indicator variables that identify each cohort plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable 
equals the difference in the mean baseline characteristic for lottery winners and other P-TECH 9-14 offerees. The 
value for other P-TECH 9-14 offerees equals the corresponding value for P-TECH 9-14 lottery winners minus the 
estimated difference between lottery winners and other P-TECH 9-14 offerees. 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 

An F test was used to assess the statistical significance of the overall difference between lottery winners and 
control group members reflected by the full set of baseline characteristics in the table. The resulting p-value is 
statistically significant (p-value <.0001). 

aOther P-TECH offerees includes students who won competitive lotteries for P-TECH 9-14 programs which 
were not their first lottery competed in, and students who matched to P-TECH 9-14 programs without going 
through a competitive lottery. 

bMedian household income has been z-scored to standardize across census tract and cohort year. 

¢Students scoring at proficiency levels 1 and 2 are not considered to be performing at grade level for state 
math and reading exams. Due to missing test scores, the sum of levels 1-4 may not add to 100 percent. 


error that is clustered by the school that students entered after their lottery. The coefficient Bo 
identifies the effect of winning a lottery on student outcomes. The standard error of By and the 
associated t-statistic identify statistical significance.* Standard errors were clustered by student’s 
grade 9 school. 


In addition to the ITT analysis, Complier Average Causal Estimate (CACE) analyses 
were also conducted, in which the estimates were scaled by the number of compliers in the treat- 
ment group. 


To estimate impacts for students who actually enroll in P-TECH 9-14, a two-stage least 
squares analyses (2SLS) was conducted. The 2SLS model included a separate instrument for each 
lottery. In effect, this model uses 2SLS to estimate the effect of ever enrolling in P-TECH for 
each lottery and pools resulting estimates across lotteries. This approach has been used for past 
analyses of randomized experiments and lottery-based studies.* 


The first stage will be specified as: 


E, = an T° Lig + pee Vj Ti Ti + Om * Sui + Op * Spi + Wi (2) 


4 A linear probability model was employed for both the binary and continuous outcomes. 


SAbdulkadiroglu et al. (2011); Bloom and Unterman (2013); Gennetian, Morris, Bos, and Bloom (2005); 
Ludwig and Kling (2007) 
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where Ei is a P-TECH 9-14 enrollment indicator equal to 1 if student 7 ever enrolled in 
P-TECH 9-14 and 0 otherwise, and all other terms are defined as in equation (1). Notably, as 
described, there is a separate instrument for each lottery. 


The second stage equation will be specified as: 
Yi, = Djar GG + OE + bu Sui + Oe Sri t i (3) 


where &£; equals the fitted value of the enrollment outcome from the first-stage equation, 
and e; is arandom error that is clustered by the school that students entered after their lottery. The 
estimated value of 6 is a consistent estimate of the average effect of enrolling in a P-TECH 9-14 
for target P-TECH 9-14 enrollees. 


Note, however, that the crossover rates from control to treatment were low among the 
sample of students. For each year, approximately three percent of the control sample was able to 
enroll in a P-TECH 9-14 school, as seen in Appendix Table A.5. 


The CACE estimates, which utilize a two-stage least squares model (2SLS), shown in 
Appendix Table A.6, had a robust first stage, with 42 instruments (one for each lottery) and re- 
sulted in an F-Test of > 1000, which far exceeds the threshold recommended by the What Works 
Clearinghouse for instrument strength.° 


Variables Used in Analysis 


Appendix Table A.7 provides information on how variables used in the model were 
coded. 


Missing Data Robustness Checks 


As described in the main report, there were nonrandom patterns of missing data associated with 
charter school enrollment for some variables which were systematically not reported to the district 
for students in charter schools; thus they are not included in the administrative records that were 
used for these analyses. For outcomes that were affected by these missing data, in addition to the 
impact analysis, the study team also conducted some bounding exercises, in which the highest 
and lowest possible values for each outcome were imputed for all charter school students who 
were missing outcomes data, in order to obtain high and low “bounds” around what the impact 
estimate could be. 


At the high end, all charter students were assumed to have earned the modal number of 
credits earned by students in the analysis sample, and at the low end, the assumption was that 
these same students had earned no credits in any years of high school. For the attendance out- 
comes, at the high end, all charter school students with missing data were assumed to have had 


SWhat Works Clearinghouse (2017). 


50 


Appendix Table A.5 


Active Student Sample Crossover Rates for 
P-TECH 9-14 Analytic Sample 


P-TECH Comparison 


Lottery Group 
Outcome Winners Members 
Year 1 
Compliance rate (%) 83.4 97.2 
Crossover rate (%) 16.6 2.8 
Sample size (total=2,985) 1,400 1,585 
Number of lotteries (total=42) 
Year 2 
Compliance rate (%) 83.9 96.6 
Crossover rate (%) 16.1 3.4 
Sample size (total=1 ,996) 1,013 983 
Number of lotteries (total=29) 
Year 3 
Compliance rate (%) 85.1 96.7 
Crossover rate (%) 14.9 3.3 
Sample size (total=1,067) 484 583 


Number of lotteries (total=15) 


SOURCES: MDRC's calculations use High School Application Processing 
System and New York City Department of Education (DOE) enrollment files 
from the 2012-2017 school years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means 

for all lottery winners. Values for the difference between P-TECH 9-14 
lottery winners and comparison group members are obtained from a 
regression of a given outcome on a series of indicator variables that identify 
each lottery plus an indicator variable that equals 1 for lottery winners and 0 
for lottery losers. The coefficient on the latter indicator variable equals the 
difference in the mean outcome for lottery winners and comparison group 
members. The value for comparison group members equals the 
corresponding value for P-TECH 9-14 lottery winners minus the estimated 
difference between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z- 
scored 8th-grade ELA test score, Missing rate of z-scored 8th-grade ELA 
test score, z-scored 8th-grade math test score, Missing rate of z-scored 8th- 
grade math test score). 

A two-tailed t-test was applied to the estimated difference. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Sample sizes reflect the total active student sample for each year, as 
compliance and cross-over rates are calculated based on students active 
within each year only. 
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Appendix Table A.6 


CACE Estimates for P-TECH 9-14 Analytic Sample 


Outcome 


Year 1 
Attendance (180 days) Y1 (%) 


Cumulative Total Credits Attempted Y1° 
Cumulative Total Credits Earned Y1° 
Cumulative Academic Credits Attempted Y1 
Cumulative Academic Credits Earned Y1 
Cumulative CTE/Other Credits Attempted Y1° 


Cumulative CTE/Other Credits Earned Y1° 
Cumulative Regents Attempted Y1 


Cumulative Regents Passed, score of 65+ Y1 
Passed Math Regents to CUNY standard Y1 (%) 
Passed ELA Regents to CUNY standard Y1 (%) 


Sample Size (total=3,161) 
Number of lotteries (total=42) 


Year 2 
Attendance (180 days) Y2 (%) 


Cumulative Total Credits Attempted Y2° 
Cumulative Total Credits Earned Y2° 
Cumulative Academic Credits Attempted Y2 
Cumulative Academic Credits Earned Y2 
Cumulative CTE/Other Credits Attempted Y2° 
Cumulative CTE/Other Credits Earned Y2° 
Cumulative Regents Attempted Y2 

Cumulative Regents Passed, score of 65+ Y2 
Passed Math Regents to CUNY standard Y2 (%) 
Passed ELA Regents to CUNY standard Y2 (%) 


Sample Size (total=2,164) 
Number of lotteries (total=29) 
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CACE? 
Estimated 


Difference 


2.4 
0.9 
0.6 
-0.5 
-0.7 
1.4 
1.3 
0.9 
0.2 
3.6 


8.4 


3.9 
2.3 
1.7 
-0.7 
-1.0 
3.2 
2.9 
2.0 
0.5 
3.3 


20.8 


P-TECH 
Winners 


N Obs? 


1,406 
1,414 
1,414 
1,414 
1,414 
1,414 
1,414 
1,479 
1,479 
1,479 


1,479 


1,044 
1,054 
1,054 
1,054 
1,054 
1,054 
1,054 
1,090 
1,090 
1,090 


1,090 


Comparison 
Group 


N Obs? 


1,523 
1,538 
1,538 
1,538 
1,538 
1,538 
1,538 
1,682 
1,682 
1,682 


1,682 


983 
1,000 
1,000 
1,000 
1,000 
1,000 
1,000 
1,074 
1,074 
1,074 


1,074 
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Appendix Table A.6 (continued) 


CACE P-TECH Comparison 


Estimated Winners Group 
Outcome Difference N Obs N Obs 
Year 3 
Attendance (180 days) Y3 (%) 4.6 511 624 
Cumulative Total Credits Attempted Y3° 4.8 515 640 
Cumulative Total Credits Earned Y3° 2.4 515 640 
Cumulative Academic Credits Attempted Y3 0.5 515 640 
Cumulative Academic Credits Earned Y3 -0.9 515 640 
Cumulative CTE/Other Credits Attempted Y3° 4.2 515 640 
Cumulative CTE/Other Credits Earned Y3° 3.5 515 640 
Cumulative Regents Attempted Y3 2.4 529 674 
Cumulative Regents Passed, score of 65+ Y3 0.5 529 674 
Passed Math Regents to CUNY standard Y3 (%) 8.4 529 674 
Passed ELA Regents to CUNY standard Y3 (%) 9.5 529 674 


Sample Size (total=1,203) 
Number of lotteries (total=15) 


SOURCES: MDRC's calculations use High School Application Processing System and New York 
City Department of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 
school years, as well as data from NYC DOE enrollment files from the 2012-2017 school years, and 
American Community Survey (ACS) data by census tract for median household income from the 
2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values 
for the difference between P-TECH 9-14 lottery winners and comparison group members are 
obtained from a regression of a given outcome on a series of indicator variables that identify 

each lottery plus an indicator variable that equals 1 for lottery winners and 0 for lottery losers. The 
coefficient on the latter indicator variable equals the difference in the mean outcome for lottery 
winners and comparison group members. The value for comparison group members equals the 
corresponding value for P-TECH 9-14 lottery winners minus the estimated difference between lottery 
winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test 
score, Missing rate of z-scored 8th-grade ELA test score, z-scored 8th-grade math test score, 
Missing rate of z-scored 8th-grade math test score). 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Attempted credits measures include credits for all courses for which the student received either a 
passing or failing grade (so does not include courses awarded no grade). 

Regents attempts variables only include Regents exams for which the student received a grade 
and does not include exams where the student was absent, ill, cheated, or the school 
misadministered the exam. 

aComplier Average Causal Estimate (CACE) analyses are estimates scaled by the number of 
compliers in the treatment group. 

bNumber of Observations. 

c¢Cumulative Total Credits measures include all credits attempted or earned including Physical 
Education and Functional Code credits, while Cumulative CTE/Other Credits measures exclude 
Physical Education and Functional Code credits. 
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Appendix Table A.7 


List of Covariates and Outcomes for Baseline and Impacts Models 


Variable 

Female 

Asian Race/Ethnicity 
Black Race/Ethnicity 
Hispanic Race/Ethnicity 
Other Race/Ethnicity 
White Race/Ethnicity 


Median HH income of neighborhood 


Missing Median HH income 


ELA Level 1 


ELA Level 2 


ELA Level 3 


Covariates 


Definiton 

Flag for female gender 

Flag for Asian race/ethnicity 
Flag for Black race/ethnicity 
Flag for Hispanic race/ethnicity 
Flag for Other race/ethnicity 
Flag for White race/ethnicity 


Standardized (z-scored) median household income from 
ACS data 


Missing Median Household income from ACS data 


Flag for 8th Grade ELA standardized test performance Level 1: 


Did not meet standards 


Flag for 8th Grade ELA standardized test performance Level 2: 


Partially met standards 


Flag for 8th Grade ELA standardized test performance Level 3: 


Fully met standards 


Range 

0,1 0.37 
0,1 0.07 
0,1 0.45 
0,1 0.43 
0,1 0.02 
0,1 0.03 
-1.7-4.9 -0.47 
0,1 0.03 
0,1 0.28 
0,1 0.46 
0,1 0.22 


Mean Deviation 


0.48 
0.25 
0.50 
0.50 
0.13 
0.18 


0.65 


0.17 


0.45 


0.50 


0.41 


(continued) 


cs 


Variable 
ELA Level 4 


Missing ELA test 


Math Level 1 


Math Level 2 


Math Level 3 


Math Level 4 


Missing Math test 

English Language Learner (ELL) 
Missing ELL status 

Enrolled in Charter School 


Attendance, percent of 180 days, Year 1 


Attendance, percent of 180 days, Year 2 


Appendix Table A.7 (continued) 


Covariates 


Definiton 
Flag for 8th Grade ELA standardized test performance Level 4: 
Met standards with distinction 


Flag for Missing 8th Grade ELA standardized test information 


Flag for 8th Grade Math standardized test performance Level 1: 


Did not meet standards 


Flag for 8th Grade Math standardized test performance Level 2: 


Partially met standards 


Flag for 8th Grade Math standardized test performance Level 3: 


Fully met standards 


Flag for 8th Grade Math standardized test performance Level 4: 


Met standards with distinction 

Flag for Missing 8th Grade Math standardized test information 
Flag for English language learner 

Flag for Missing English language learner status 

Flag for enrolled in a charter school in spring of 8th Grade 


Attendance rate as percent days attended out of 180 days in 
first program year 


Attendance rate as percent days attended out of 180 days in 
second program year 


Range 


0,1 


0,1 


0,1 


0,1 


0,1 


0-97.2 


Mean Deviation 


0.04 


0.04 


0.45 


0.39 


0.12 


0.04 


0.13 


0.11 


0.06 


0.09 


77.2 


72.2 


0.19 


0.20 


0.50 


0.49 


0.33 


0.19 
0.34 
0.31 
0.24 


0.29 


24.0 


27.2 
(continued) 
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Variable 
Attendance, percent of 180 days, Year 3 


Cumulative Total Credits Attempted, Year 1 


Cumulative Total Credits Attempted, Year 2 


Cumulative Total Credits Attempted, Year 3 


Cumulative Total Credits Earned, Year 1 
Cumulative Total Credits Earned, Year 2 
Cumulative Total Credits Earned, Year 3 


Cumulative Academic Credits Attempted, Year 1 


Cumulative Academic Credits Attempted, Year 2 


Cumulative Academic Credits Attempted, Year 3 


Cumulative Academic Credits Earned, Year 1 


Appendix Table A.7 (continued) 


Covariates 
Definiton 
Attendance rate as percent days attended out of 180 days in 
third program year 


Total credits attempted, cumulative through first 
program year 


Total credits attempted, cumulative through second 
program year 


Total credits attempted, cumulative through third 
program year 


Total credits earned, cumulative through first program year 
Total credits earned, cumulative through second program year 
Total credits earned, cumulative through third program year 


Academic credits attempted, cumulative through first 
program year 


Academic credits attempted, cumulative through second 
program year 


Academic credits attempted, cumulative through third 
program year 


Academic credits earned, cumulative through first 
program year 


Range 


0 - 97.8 


0 - 25.2 


0 - 46.1 


Mean Deviation 


67.3 31.0 


13.4 4.5 


26.9 8.8 


39.8 14.1 
11.4 4.9 
21.9 9.7 


31.8 14.7 


98 3.7 


19.7 6.7 


29.2 10.6 


8.2 3.9 
(continued) 
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Variable 
Cumulative Academic Credits Earned, Year 2 


Cumulative Academic Credits Earned, Year 3 


Cumulative CTE/Other Credits Attempted, Year 1 


Cumulative CTE/Other Credits Attempted, Year 2 


Cumulative CTE/Other Credits Attempted, Year 3 


Cumulative CTE/Other Credits Earned, Year 1 


Cumulative CTE/Other Credits Earned, Year 2 


Cumulative CTE/Other Credits Earned, Year 3 


Cumulative Regents Exams Attempted, Year 1 


Cumulative Regents Exams Attempted, Year 2 


Appendix Table A.7 (continued) 


Covariates 
Definiton 
Academic credits earned, cumulative through second 
program year 


Academic credits earned, cumulative through third 
program year 


CTE/Other credits attempted, cumulative through first 
program year 


CTE/Other credits attempted, cumulative through second 
program year 


CTE/Other credits attempted, cumulative through third 
program year 


CTE/Other credits earned, cumulative through first 
program year 


CTE/Other credits earned, cumulative through second 
program year 


CTE/Other credits earned, cumulative through third 
program year 


Total Regents exams attempted, cumulative through first 
program year 


Total Regents exams attempted, cumulative through second 
program year 


Mean Deviation 


15.7 7.4 
22.8 11.0 
2.3 1.8 
4.7 3.1 
68 4.6 
2.1 1.8 
4.1 3.0 
5.9 4.4 
2.2 1.7 
5.5 3.4 


(continued) 
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Appendix Table A.7 (continued) 


Covariates 

Variable Definiton Range Mean Deviation 
Cumulative Regents Exams Attempted, Year 3 Total Regents exams attempted, cumulative through third 

program year 0 - 34 8.5 5.2 
Cumulative Regents Exams Passed, Year 1 Total Regents exams passed (score of 65+), cumulative 

through first program year 0-7 1.2 1.2 
Cumulative Regents Exams Passed, Year 2 Total Regents exams passed (score of 65+), cumulative 

through second program year 0-13 27 2.2 
Cumulative Regents Exams Passed, Year 3 Total Regents exams passed (score of 65+), cumulative 

through third program year 0-17 40 3.0 
Passed Math Regents to CUNY Standard, Year 1 Flag for passed 1+ Math Regents to the CUNY admission 

standard, cumulative through first program year 0,1 0.31 0.46 
Passed Math Regents to CUNY Standard, Year 2 Flag for passed 1+ Math Regents to the CUNY admission 

standard, cumulative through second program year 0,1 0.41 0.49 
Passed Math Regents to CUNY Standard, Year 3 Flag for passed 1+ Math Regents to the CUNY admission 

standard, cumulative through third program year 0,1 0.45 0.50 
Passed ELA Regents to CUNY Standard, Year 1 Flag for passed 1+ ELA Regents to the CUNY admission 

standard, cumulative through first program year 0,1 0.06 0.24 
Passed ELA Regents to CUNY Standard, Year 2 Flag for passed 1+ ELA Regents to the CUNY admission 

standard, cumulative through second program year 0,1 0.31 0.46 
Passed ELA Regents to CUNY Standard, Year 3 Flag for passed 1+ ELA Regents to the CUNY admission 

standard, cumulative through third program year 0,1 0.52 0.50 


(continued) 
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Variable 
Newly enrolled in a Charter School, Year 1 


Newly enrolled in a Charter School, Year 2 


Newly enrolled in a Charter School, Year 3 


Continuing enrollment in a Charter School, Year 2 


Continuing enrollment in a Charter School, Year 3 


Inactive, Year 1 


Inactive, Year 2 


Inactive, Year 3 


Missing Attendance, percent of 180 days, Year 1 


Missing Attendance, percent of 180 days, Year 2 


Missing Attendance, percent of 180 days, Year 3 


Appendix Table A.7 (continued) 


Covariates 
Definiton 
Flag for newly enrolled in a charter school in first year in 
the program 


Flag for newly enrolled in a charter school in second year in 
the program 


Flag for newly enrolled in a charter school in third year in 
the program 


Flag for continuous enrollment a charter school by second year 
in the program 


Flag for continuous enrollment a charter school by third year 
in the program 


Flag for inactive status in NYC DOE in first year in the program 


Flag for inactive status in NYC DOE in second year 
in the program 


Flag for inactive status in NYC DOE in third year 
in the program 


Flag for missing attendance rate as percent days attended out 
of 180 days in first program year 


Flag for missing attendance rate as percent days attended out 
of 180 days in second program year 


Flag for missing attendance rate as percent days attended out 
of 180 days in third program year 


Range 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


Mean Deviation 


0.08 


0.01 


0.00 


0.06 


0.05 


0.06 


0.08 


0.11 


0.12 


0.14 


0.16 


0.26 


0.08 


0.07 


0.23 


0.22 


0.23 


0.27 


0.32 


0.33 


0.34 


0.37 
(continued) 
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Variable 
Missing Cumulative Total Credits Attempted, Year 1 


Missing Cumulative Total Credits Attempted, Year 2 


Missing Cumulative Total Credits Attempted, Year 3 


Missing Cumulative Total Credits Earned, Year 1 


Missing Cumulative Total Credits Earned, Year 2 


Missing Cumulative Total Credits Earned, Year 3 


Missing Cumulative Academic Credits Attempted, Year 1 


Missing Cumulative Academic Credits Attempted, Year 2 


Missing Cumulative Academic Credits Attempted, Year 3 


Missing Cumulative Academic Credits Earned, Year 1 


Missing Cumulative Academic Credits Earned, Year 2 


Appendix Table A.7 (continued) 


Covariates 
Definiton 
Flag for missing total credits attempted, cumulative through 
first program year 


Flag for missing total credits attempted, cumulative through 
second program year 


Flag for missing total credits attempted, cumulative through 
third program year 


Flag for missing total credits earned, cumulative through 
first program year 


Flag for missing total credits earned, cumulative through 
second program year 


Flag for missing total credits earned, cumulative through 
third program year 


Flag for missing academic credits attempted, cumulative 
through first program year 


Flag for missing academic credits attempted, cumulative 
through second program year 


Flag for missing academic credits attempted, cumulative 
through third program year 


Flag for missing academic credits earned, cumulative 
through first program year 


Flag for missing academic credits earned, cumulative 
through second program year 


Range 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


Mean Deviation 


0.12 


0.11 


0.10 


0.12 


0.11 


0.11 


0.12 


0.11 


0.10 


0.12 


0.11 


0.33 


0.31 


0.30 


0.33 


0.31 


0.31 


0.33 


0.31 


0.30 


0.33 


0.31 
(continued) 
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Variable 
Missing Cumulative Academic Credits Earned, Year 3 


Missing Cumulative CTE/Other Credits Attempted, Year 1 


Missing Cumulative CTE/Other Credits Attempted, Year 2 


Missing Cumulative CTE/Other Credits Attempted, Year 3 


Missing Cumulative CTE/Other Credits Earned, Year 1 


Missing Cumulative CTE/Other Credits Earned, Year 2 


Missing Cumulative CTE/Other Credits Earned, Year 3 


Missing Cumulative Regents Exams Attempted, Year 1 


Missing Cumulative Regents Exams Attempted, Year 2 


Missing Cumulative Regents Exams Attempted, Year 3 


Appendix Table A.7 (continued) 


Covariates 
Definiton 
Flag for missing academic credits earned, cumulative 
through third program year 


Flag for missing CTE/Other credits attempted, cumulative 
through first program year 


Flag for missing CTE/Other credits attempted, cumulative 
through second program year 


Flag for missing CTE/Other credits attempted, cumulative 
through third program year 


Flag for missing CTE/Other credits earned, cumulative 
through first program year 


Flag for missing CTE/Other credits earned, cumulative 
through second program year 


Flag for missing CTE/Other credits earned, cumulative 
through third program year 


Flag for missing total Regents exams attempted, cumulative 
through first program year 


Flag for missing total Regents exams attempted, cumulative 
through second program year 


Flag for missing total Regents exams attempted, cumulative 
through third program year 


Range 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


0,1 


Mean Deviation 


0.11 


0.13 


0.11 


0.12 


0.13 


0.12 


0.12 


0.00 


0.00 


0.00 


0.31 


0.33 


0.32 


0.32 


0.33 


0.32 


0.33 


0.00 


0.00 


0.00 
(continued) 
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Variable 
Missing Cumulative Regents Exams Passed, Year 1 


Missing Cumulative Regents Exams Passed, Year 2 


Missing Cumulative Regents Exams Passed, Year 3 


Appendix Table A.7 (continued) 


Covariates 
Definiton Range Mean Deviation 
Flag for missing total Regents exams passed (score of 65+), 
cumulative through first program year 0,1 0.00 0.00 


Flag for missing total Regents exams passed (score of 65+), 
cumulative through second program year 0,1 0.00 0.00 


Flag for missing total Regents exams passed (score of 65+), 
cumulative through third program year 0,1 0.00 0.00 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department of Education (NYC DOE) state test data for 
eighth-graders from the 2012-2016 school years, as well as data from NYC DOE enrollment files from the 2012-2017 school years, and American Community 
Survey (ACS) data by census tract for median household income from the 2012-2017 calendar years. 


the average attendance rate for those schools, which is published by the New York State Edu- 
cation Department’ and which was matched at the school level; at the low end, it was assumed 
that these students never attended school. For the CTE course-credit outcomes, at the high end, 
the charter students were assumed to have earned the modal value of these courses taken by 
program group students, and at the low end it was assumed that charter school students took no 
CTE courses. 


For outcomes where the high-bound estimate was not statistically significant but the 
standard estimate (that is, the estimate with the missing outcomes data) was, for purposes of dis- 
cussion in the main text the result was not considered significant, and the results in the tables in 
the main text include a symbol to flag these results. However, in cases where both the high-bound 
estimate and the standard estimate are both significant, the result is considered significant. 


Statistically significant differences in missing data rates for students in program and com- 
parison groups were not found for the Year 1 sample. In the Year 2 sample there are statistically 
significant differences in missing data rates for attendance, and in both academic and CTE and 
other nonacademic credits. Specifically, for attendance, 4.7 percentage points more comparison 
group students were missing attendance data than students in the program group. For the credit 
outcomes, the difference was 4.4 percentage points. For the Year 3 sample across groups there 
was a Statistically significant difference in missing attendance data rates of 7.8 percentage points; 
according to the What Works Clearinghouse Handbook, tolerable levels of missing data for 15 
percent attrition or less are between five and six percent. However, the figures discussed here 
(Appendix Table A.8) are for missing data rates for students who remained in New York City 
public schools of some kind (that is, regular public and charter), and should be combined with the 
“imactive rates” for each sample reported in the main text, for purposes of understanding the po- 
tential for attrition bias.* 


Results of the bounding exercises are presented in Appendix Table A.9. 


Additional Regents Exam Analysis 


The main narrative presents analyses of Regents exam taking that includes the following out- 
comes: the number of Regents exams taken and passed with a 65 or higher; and the percentage 
of students who pass the Regents math and ELA exams with a CUNY-qualifying score. Addi- 
tional analyses were also conducted that show the number of exams both taken and passed in each 
of the three follow-up years, by math and ELA subject. Results are displayed in Appendix Table 
A.10. 


Attendance data were sourced from the annual School Report Card database. NYSED (2019) 
8What Works Clearinghouse (2017) 
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Appendix Table A.8 


Outcome Impacts Missing Rates for P-TECH 9-14 First Lottery Participants: 
Intent to Treat Sample 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 

Outcome Winners Members _ Difference Difference Difference 
Year 1 
Missing Attendance (180 days) Y1 (%) 9.4 13.1 -3.7 -0.11 0.154 
Missing Cumul.Total Credits 
Attempted Y1 (%) 9.7 13.7 -4.0 -0.11 0.123 
Missing Cumul. Total Credits 
Earned Y1 (%) 9.7 13.7 -4.0 -0.11 0.123 
Missing Cumul. Academic Credits 
Attempted Y1 (%) 9.7 13.7 -4.0 -0.11 0.123 
Missing Cumul. Academic Credits 
Earned Y1 (%) 9.7 13.7 -4.0 -0.11 0.123 
Missing Cumul. CTE/Other Credits 
Attempted Y1 (%) 9.7 13.7 -4.0 -0.11 0.123 
Missing Cumul. CTE/Other Credits 
Earned Y1 (%) 9.7 13.7 -4.0 -0.11 0.123 
Missing Cumul. Regents 
Attempted Y1 (%) 0.0 0.0 0.0 
Missing Cumul. Regents Passed, 
score of 65+ Y1 (%) 0.0 0.0 0.0 
Sample size (total=3,161) 1,479 1,682 


Number of lotteries (total=42) 
(continued) 


CTE Course Subjects and Coding 


Appendix Table A.11 offers descriptive analysis of the CTE and other nonacademic 
courses that students in the program and comparison groups enrolled in, by subject. 
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Appendix Table A.8 (continued) 


P-TECH Comparison 


Lottery Group 
Outcome Winners Members 
Year 2 
Missing Attendance (180 days) Y2 (%) 11.0 15.7 
Missing Cumul.Total Credits 
Attempted Y2 (%) 8.3 12.8 
Missing Cumul. Total Credits 
Earned Y2 (%) 8.3 12.8 
Missing Cumul. Academic Credits 
Attempted Y2 (%) 8.3 12.8 
Missing Cumul. Academic Credits 
Earned Y2 (%) 8.3 12.8 
Missing Cumul. CTE/Other Credits 
Attempted Y2 (%) 8.3 12.8 
Missing Cumul. CTE/Other Credits 
Earned Y2 (%) 8.3 12.8 
Missing Cumul. Regents 
Attempted Y2 (%) 0.0 0.0 
Missing Cumul. Regents Passed, 
score of 65+ Y2 (%) 0.0 0.0 
Sample size (total=2,164) 1,090 1,074 


Number of lotteries (total=29) 


65 


Estimated 
Difference 


-4.7 * 


-44™ 


-44™ 


-44™ 


-44™ 


-44™ 


44" 


0.0 


0.0 


Effect Size of P-Value for 


Estimated 
Difference 


-0.13 


-0.13 


-0.13 


-0.13 


-0.13 


-0.13 


-0.13 


Estimated 
Difference 


0.068 


0.046 


0.046 


0.046 


0.046 


0.046 


0.046 
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P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 
Outcome Winners Members _ Difference Difference Difference 
Year 3 
Missing Attendance (180 days) Y3 (%) 11.2 18.9 -7.8 *** -0.19 0.007 
Missing Cumul. Total Credits 
Attempted Y3 (%) 7.6 11.4 -3.9 -0.12 0.125 
Missing Cumul. Total Credits 
Earned Y3 (%) 7.6 11.4 -3.9 -0.12 0.125 
Missing Cumul. Academic Credits 
Attempted Y3 (%) 7.6 11.4 -3.9 -0.12 0.125 
Missing Cumul. Academic Credits 
Earned Y3 (%) 7.6 11.4 -3.9 -0.12 0.125 
Missing Cumul. CTE/Other Credits 
Attempted Y3 (%) 7.6 11.4 -3.9 -0.12 0.125 
Missing Cumul. CTE/Other Credits 
Earned Y3 (%) 7.6 11.4 -3.9 -0.12 0.125 
Missing Cumul. Regents 
Earned Y3 (%) 0.0 0.0 0.0 
Missing Cumul. Regents Passed, 
score of 65+ Y3 (%) 0.0 0.0 0.0 
Sample size (total=1,203) 529 674 


Number of lotteries (total=15) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as 
well as data from NYC DOE enrollment files from the 2012-2017 school years, and American Community 
Survey (ACS) data by census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a 
regression of a given outcome on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable 
equals the difference in the mean outcome for lottery winners and comparison group members. The value for 
comparison group members equals the corresponding value for P-TECH 9-14 lottery winners minus the 
estimated difference between lottery winners and comparison group members. 


Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test score, Missing 
rate of z-scored 8th-grade ELA test score, z-scored 8th-grade math test score, Missing rate of z-scored 8th- 
grade math test score). 


A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 
1 percent; ** = 5 percent; * = 10 percent. 
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Impacts for P-TECH 9-14 Analytic Sample: Bounded Estimates 


Outcome 


Year 1 

Attendance (180 days) Y1 (%) 
High imputed estimate 
Low imputed estimate 


Cumulative Total Credits Attempted Y1 : 
High imputed estimate 


Low imputed estimate 


Cumulative Total Credits Earned Y1° 
High imputed estimate 
Low imputed estimate 


Cumulative Academic Credits Attempted Y1 
High imputed estimate 
Low imputed estimate 


Cumulative Academic Credits Earned Y1 
High imputed estimate 
Low imputed estimate 


Cumulative CTE/Other Credits Attempted Y1 2 
High imputed estimate 


Low imputed estimate 


Cumulative CTE/Other Credits Earned Y1° 
High imputed estimate 


Low imputed estimate 


Sample size (total=3,161) 
Number of lotteries (total=42) 


Appendix Table A.9 


P-TECH Comparison 


Lottery 
Winners 


78.5 
79.3 
74.9 


13.7 
13.7 
13.1 


11.7 
11.8 
11.2 


9.6 
9.6 
9.2 


8.1 
8.2 
7.7 


2.7 
2.7 
2.6 


2.5 
2.5 
2.4 


1,479 


Group 


Members 


67 


76.6 
78.2 
70.2 


12.9 
13.0 
11.8 


11.2 
11.4 
10.3 


10.0 
10.0 
9.2 


8.6 
8.7 
7.9 


1.6 
1.6 
1.5 


1.4 
1.5 
1.3 


1,682 


Estimated 
Difference 


1.9 * 
1.2 
4.7 * 


0.8 * 
0.7 * 
1.2 ** 


0.5 
0.4 
0.9 * 


-0.4 
-0.4 
0.0 


-0.5 
-0.6 
-0.2 


1.1 KK 
1.1 KK 
1.1 KK 


1.1 RK 
1.0 RK 
1.1 KK 


P-Value for P-TECH Comparison 
Estimated Winners 


Difference 


0.082 
0.216 
0.056 


0.059 
0.077 
0.045 


0.113 
0.151 
0.061 


0.424 
0.468 
0.998 


0.148 
0.145 
0.669 


0.002 
0.004 
0.003 


0.001 
0.002 
0.002 


N Obs? 


1,406 
1,474 
1,474 


1,414 
1,479 
1,479 


1,414 
1,479 
1,479 


1,414 
1,479 
1,479 


1,414 
1,479 
1,479 


1,414 
1,479 
1,479 


1,414 
1,479 
1,479 


Group 
N Obs? 


1,523 
1,670 
1,670 


1,538 
1,682 
1,682 


1,538 
1,682 
1,682 


1,538 
1,682 
1,682 


1,538 
1,682 
1,682 


1,538 
1,682 
1,682 


1,538 
1,682 
1,682 


(continued) 


Appendix Table A.9 (continued) 
P-TECH Comparison 


Outcome 

Year 2 

Attendance (180 days) Y2 (%) 
High imputed estimate 
Low imputed estimate 


Cumulative Total Credits Attempted y2? 
High imputed estimate 
Low imputed estimate 


Cumulative Total Credits Earned Y2° 
High imputed estimate 
Low imputed estimate 


Cumulative Academic Credits Attempted Y2 
High imputed estimate 
Low imputed estimate 


Cumulative Academic Credits Earned Y2 
High imputed estimate 
Low imputed estimate 


Cumulative CTE/Other Credits Attempted y2? 
High imputed estimate 
Low imputed estimate 


Cumulative CTE/Other Credits Earned Y2° 
High imputed estimate 


Low imputed estimate 


Sample size (total=2,164) 
Number of lotteries (total=29) 


Lottery 
Winners 


74.3 
75.1 
71.6 


27.6 
27.2 
26.7 


22.8 
22.5 
22.0 


19.6 
19.6 
18.9 


15.8 
15.6 
15.2 


5.6 
5.5 
5.4 


5.0 
4.9 
4.8 


1,090 


Group 


Members 


68 


71.2 
73.1 
65.2 


25.7 
24.9 
23.9 


21.4 
20.9 
19.8 


20.2 
20.1 
18.7 


16.5 
16.1 
15.3 


3.1 
3.0 
2.8 


2.7 
2.6 
2.5 


1,074 


Estimated 
Difference 


3.1 KKK 
200" 
pa a 


1.9 ** 
2.3 ™* 
2.8 ** 


1.3 ™ 
1.6 ™* 
2.2 ** 


-0.6 
-0.6 
0.2 


-0.8 
-0.5 
-0.1 


26 RK 
25 KK 
2.6 KK 


2.3 KKK 
2.3 kK 
2.3 KK 


P-Value for P-TECH Comparison 
Estimated Winners 


Difference 


0.004 
0.047 
0.002 


0.031 
0.020 
0.015 


0.039 
0.023 
0.015 


0.419 
0.434 
0.780 


0.130 
0.336 
0.862 


<.0001 
<.0001 
<.0001 


<.0001 
<.0001 
<.0001 


N Obs 


1,044 
1,084 
1,084 


1,054 
1,090 
1,090 


1,054 
1,090 
1,090 


1,054 
1,090 
1,090 


1,054 
1,090 
1,090 


1,054 
1,090 
1,090 


1,054 
1,090 
1,090 


Group 
N Obs 


983 
1,069 
1,069 


1,000 
1,074 
1,074 


1,000 
1,074 
1,074 


1,000 
1,074 
1,074 


1,000 
1,074 
1,074 


1,000 
1,074 
1,074 


1,000 
1,074 
1,074 
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Appendix Table A.9 (continued) 
P-TECH Comparison 


Outcome 

Year 3 

Attendance (180 days) Y3 (%) 
High imputed estimate 
Low imputed estimate 


Cumulative Total Credits Attempted Y3? 
High imputed estimate 
Low imputed estimate 


Cumulative Total Credits Earned Y3° 
High imputed estimate 


Low imputed estimate 


Cumulative Academic Credits Attempted Y3 
High imputed estimate 
Low imputed estimate 


Cumulative Academic Credits Earned Y3 
High imputed estimate 
Low imputed estimate 


Cumulative CTE/Other Credits Attempted Y3? 
High imputed estimate 
Low imputed estimate 


Cumulative CTE/Other Credits Earned Y3° 
High imputed estimate 
Low imputed estimate 


Sample size (total=1,203) 
Number of lotteries (total=15) 


Lottery 
Winners 


70.6 
71.5 
68.5 


42.2 
41.5 
41.1 


33.6 
33.1 
32.7 


30.0 
29.4 
29.2 


23.2 
22.8 
22.6 


8.4 
8.3 
8.2 


7.3 
7.1 
7.1 


529 


Group 


Members 


69 


66.8 
69.0 
62.1 


38.3 
36.8 
36.0 


31.6 
30.5 
29.7 


29.6 
28.4 
27.8 


24.0 
23.1 
22.5 


5.0 
4.9 
4.6 


4.4 
4.2 
4.1 


674 


Estimated 
Difference 


a5 
2.5 
pa a 


4.0. 
Os Bic 
5.1 KK 


20% 
oe 
a0 


0.4 
1.0 
1.4 


-0.8 
-0.3 
0.1 


g4 
3.4 
3.6 RK 


2.8 RKK 
29 kKK 
29 KK 


P-Value for P-TECH Comparison 
Estimated Winners 


Difference 


0.037 
0.171 
0.006 


0.004 
0.003 
0.004 


0.023 
0.010 
0.010 


0.733 
0.407 
0.301 


0.295 
0.711 
0.937 


<.0001 
<.0001 
<.0001 


<.0001 
<.0001 
0.000 


N Obs 


511 
527 
527 


515 
529 
529 


515 
529 
529 


515 
529 
529 


515 
529 
529 


515 
529 
529 


515 
529 
529 


Group 
N Obs 


624 
666 
666 


640 
674 
674 


640 
674 
674 


640 
674 
674 


640 
674 
674 


640 
674 
674 


640 
674 
674 
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Appendix Table A.9 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City Department of 
Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as well as data from NYC DOE 
enrollment files from the 2012-2017 school years, and American Community Survey (ACS) data by census tract for median 
household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the difference 
between P-TECH 9-14 lottery winners and comparison group members are obtained from a regression of a given outcome on 
a series of indicator variables that identify each lottery plus an indicator variable that equals 1 for lottery winners and 0 for 
lottery losers. The coefficient on the latter indicator variable equals the difference in the mean outcome for lottery winners and 
comparison group members. The value for comparison group members equals the corresponding value for P-TECH 9-14 
lottery winners minus the estimated difference between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th Grade ELA test score, Missing rate of z- 
scored 8th-grade ELA test score, z-scored 8th-grade math test score, Missing rate of z-scored 8th-grade math test score). 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** = 1 percent; ** 
= 5 percent; * = 10 percent. 

Attempted credits measures include credits for all courses for which the student received either a passing or failing grade 
(so does not include courses awarded no grade). 

High imputed estimates: Imputed estimates were created for active charter school attendees. For Attendance, school-level 
mean attendance values were imputed where available, else values of 100% were imputed. For all credits variables, the modal 
values among records with data were imputed. 

Low imputed estimates: Imputed estimates were created for active charter school attendees. For Attendance, values of 0% 
were imputed. For all credits variables, 0 values were imputed. 

aNumber of Observations. 

>Cumulative Total Credits measures include all credits attempted or earned including Physical Education and Functional 
Code credits, while Cumulative CTE/Other Credits measures exclude Physical Education and Functional Code credits. 
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Appendix Table A.10 


Regents Attempted and Earned by Subject: Impacts for 
P-TECH 9-14 Analytic Sample 


P-TECH Comparison Effect Size of P-Value for 
Lottery Group Estimated Estimated Estimated 
Outcome Winners Members _ Difference Difference Difference 
Year 1 
Cumulative Math Regents Attempted Y1 1.3 0.9 0.4 *** 0.58 0.001 
Cumulative Math Regents Passed, score 
of 65+ Y1 0.6 0.5 0.1 0.18 0.102 
Cumulative ELA Regents Attempted Y1 0.4 0.0 0.4 * 1.07 0.056 
Cumulative ELA Regents Passed, score 
of 65+ Y1 0.2 0.0 0.2 ** 0.69 0.050 
Sample size (total=3,161) 1,479 1,682 
Number of lotteries (total=42) 
Year 2 
Cumulative Math Regents Attempted Y2 2.5 1.8 0.6 *** 0.50 <.0001 
Cumulative Math Regents Passed, score 
of 65+ Y2 0.9 0.9 0.1 0.11 0.191 
Cumulative ELA Regents Attempted Y2 fled 0.4 0.9 *** 1.36 <.0001 
Cumulative ELA Regents Passed, score 
of 65+ Y2 0.8 0.3 0.5 *** 0.87 <.0001 
Sample size (total=2,164) 1,090 1,074 
Number of lotteries (total=29) 
Year 3 
Cumulative Math Regents Attempted Y3 3.5 2.6 0.9 *** 0.53 <.0001 
Cumulative Math Regents Passed, score 
of 65+ Y3 1.3 1.1 0.2 ** 0.24 0.023 
Cumulative ELA Regents Attempted Y3 2.0 1.0 1.0; 5°" 1.05 0.001 
Cumulative ELA Regents Passed, score 
of 65+ Y3 1.1 0.7 0.4 *** 0.64 0.001 
Sample size (total=1,203) 529 674 
Number of lotteries (total=15) 
(continued) 
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Appendix Table A.10 (continued) 


SOURCES: MDRC's calculations use High School Application Processing System and New York City 
Department of Education (NYC DOE) state test data for eighth-graders from the 2012-2016 school years, as 
well as data from NYC DOE enrollment files from the 2012-2017 school years, and American Community 
Survey (ACS) data by census tract for median household income from the 2012-2017 calendar years. 


NOTES: Values for P-TECH 9-14 lottery winners are the simple means for all lottery winners. Values for the 
difference between P-TECH 9-14 lottery winners and comparison group members are obtained from a 
regression of a given outcome on a series of indicator variables that identify each lottery plus an indicator 
variable that equals 1 for lottery winners and 0 for lottery losers. The coefficient on the latter indicator variable 
equals the difference in the mean outcome for lottery winners and comparison group members. The value for 
comparison group members equals the corresponding value for P-TECH 9-14 lottery winners minus the 
estimated difference between lottery winners and comparison group members. 

Model includes adjustment for select baseline covariates (Female, z-scored 8th-grade ELA test score, 
Missing rate of z-scored 8th-grade ELA test score, z-scored 8th-grade math test score, Missing rate of z-scored 
8th-grade math test score). 

A two-tailed t-test was applied to the estimated difference. Statistical significance levels are indicated as: *** 
= 1 percent; ** = 5 percent; * = 10 percent. 

Sample sizes are from the ITT random assignment sample; each measure's data availability can be found in 
Table A.8. 

Regents attempts variables only include Regents exams for which the student received a grade and does 
not include exams where the student was absent, ill, cheated, or the school misadministered the exam. 
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Appendix Table A.11 


P-TECH 9-14 First Lottery Participants: 
Full Category List of Treatment/Comparison 
Differences in CTE/Other Courses Attempted 


P-TECH Comparison 


Lottery Group 
Other/CTE Course Category Winners Members 
Advisory 0.1% 0.4% 
Architecture / Civil Engineering 1.1% 0.3% 
Arts 14.7% 21.3% 
Business 2.7% 2.5% 
College and Career Exploration 0.8% 0.9% 
Electrical 0.1% 0.3% 
Engineering - General 3.1% 0.5% 
Guidance 11.9% 13.0% 
Human Services 5.4% 2.0% 
Other CTE 0.1% 0.7% 
Physical Education & Health 41.4% 50.9% 
Technology 9.8% 6.0% 
Trades 0.1% 0.4% 
Undefined/Functional Codes 1.3% 0.5% 
Work-Based Learning 7.5% 0.2% 
Total Other/CTE Courses 100% 100% 


SOURCES: MDRC's calculations use High School Application Processing 
System and New York City Department of Education (NYC DOE) enrollment 
files from the 2013-2017 school years. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of 
social and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los 
Angeles, MDRC is best known for mounting rigorous, large-scale, real-world tests of new and 
existing policies and programs. Its projects are a mix of demonstrations (field tests of promising 
new program approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research and organizational 
experience to their work, providing expertise on the latest in qualitative and quantitative 
methods and on program design, development, implementation, and management. MDRC 
seeks to learn not just whether a program is effective but also how and why the program’s 
effects occur. In addition, it tries to place each project’s findings in the broader context of 
related research — in order to build knowledge about what works across the social and 
education policy fields. MDRC’s findings, lessons, and best practices are shared with a broad 
audience in the policy and practitioner community as well as with the general public and the 
media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs for ex- 
prisoners, and programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 


e Promoting Family Well-Being and Children’s Development 
e Improving Public Education 

e Raising Academic Achievement and Persistence in College 
e Supporting Low-Wage Workers and Communities 

e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local 
governments, public school systems, community organizations, and numerous private 
philanthropies. 


